Anthropic Fixes robots.txt Error After Claude AI Session Links Indexed by Search Engines

Anthropic's Claude AI session sharing links were misconfigured, leading to a large number of user conversations containing private content being crawled by search engines such as Google. Over the weekend, users discovered that Claude AI session shares appeared in Google search results. The cause was that Anthropic had not set the sharing links to prohibit search engine crawling, so using the SITE command could query a large number of session sharing links already indexed by search engines. Some of these session sharing links contained users' private information, such as wallet private keys, company task details, and personally identifiable content.

The links indexed by search engines were primarily those actively generated by users; anyone who knew the link could publicly access the conversation.

The root cause was a technical configuration error: the /share/* path in Claude's robots. txt was set to disallow crawling, but the page response header also set the X-Robots-Tag: none attribute, which is equivalent to noindex, nofollow.

Anthropic has now modified its policy to prohibit search engine crawling. A similar issue occurred with ChatGPT in 2025, when OpenAI also failed to set strict anti-crawler restrictions. Baidu Netdisk experienced a similar situation, where publicly shared links without passwords could be crawled by search engine spiders, exposing user files.