I've Been Deindexed by Google
Summary
博主分享如何成功让自己的网站被 Google 除名(deindex),以应对 AI 抓取带来的内容被肆意爬取的问题,并解释了 robots.txt 与 meta robots 标签配合使用的技巧。
View Cached Full Text
Cached at: 10/08/26, 12:21 PM
Similar Articles
I opened my small website to AI agents and logged who showed up. Mostly it’s bots indexing bots.
一位小网站站长开放了对 AI 代理的支持(llms.txt、Markdown、MCP 服务器、按次付费端点)并记录流量:绝大多数是 Meta、Amazon 等 AI 爬虫以及目录和扫描类 MCP 客户端,真正干活的代理调用仅有 3 次,说明当前的“代理互联网”主要是机器人在索引机器人。
Hi everyone, I’d like to ask: how can I get my website successfully indexed by AI?
The user inquires about methods to get their website indexed by AI search engines to enhance SEO.
@YRSM_Simon: Many people say AI Overview is just copying Google's top ten. Rank high, get cited more. Others say doing GEO is a pseudoscience — you have to learn a bunch of jargon to get cited by AI. I don't believe it. With a skeptical attitude, I took 1,320 keywords and 9,509 AI citations and checked them one by one against the...
An analysis based on 9,509 AI citations, pointing out that Google AI Overview does not simply copy the top ten results. Instead, it prefers videos, lists, and official product sites, and small and medium-sized sites also have a chance to get cited. GEO is essentially the same as SEO.
He Manipulated AI Search With 50 Articles (Exposing GEO/AEO)
SEO operator Kasra Dash showed that 50 self-referencing listicles reliably hijacked rankings inside ChatGPT, Claude, Gemini, Perplexity, Grok and Google AI Overviews without backlinks, and the URLs kept being cited even after deletion.
@huoshan007: I dug up a beast. crawl4ai, GitHub's top crawler, turned one guy's rage from getting scammed by a $16 paid service stra…
推文介绍了GitHub上爆红的开源爬虫项目crawl4ai(51k星),可将任意网站内容转为LLM可直接使用的干净Markdown,无需API密钥或付费订阅,被认为打破了付费爬虫服务的价值壁垒。