Claude-SearchBot, Anthropic's crawler for search
Claude-SearchBot is the crawler Anthropic uses to improve the quality of search results for Claude's users. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your site's visibility in user search results.
Checked against Anthropic's own documentation on October 13, 2026.
What Claude-SearchBot is for.
Anthropic says Claude-SearchBot navigates the web to improve search result quality for users, analyzing online content to make search responses more relevant and accurate.[1]
Run by
Anthropic
Its job
Search index. It crawls on its own schedule, to find pages for search results.
Its name in robots.txt
Claude-SearchBotDoes it follow robots.txt?
Yes, by its operator's account. Anthropic says its bots honor industry standard directives in robots.txt, and that it supports the non-standard Crawl-delay extension.[1]
Its IP ranges
Anthropic publishes one list of source IP addresses for its bots: an address on it indicates the crawler is coming from Anthropic. It adds that blocking those addresses may not work as an opt out, because it stops its bots from reading your robots.txt.[1] claude.com/crawling/bots.json
Every subdomain needs its own lines
Anthropic asks you to add the lines to the robots.txt of every subdomain you want to opt out.[1]
It doesn't get past CAPTCHAs
Anthropic says its bots will not attempt to bypass CAPTCHAs on the sites they crawl.[1]
No published user agent string for Claude-SearchBot.
Anthropic's page gives Claude-SearchBot as the name to match in robots.txt and publishes no full user agent string for it.[1]
The lines that allow and block Claude-SearchBot.
Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to Claude-SearchBot now.
Allow Claude-SearchBot
With no robots.txt, or one that doesn't disallow it, Claude-SearchBot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the
User-agent: *group only when it finds none.[2]User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.[1]
User-agent: Claude-SearchBot Disallow: /
How many cited sites block Claude-SearchBot.
We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.
of cited sites close their homepage to Claude-SearchBot in robots.txt: 12 of 1,529.
Blue: Claude-SearchBot. Gray: the other 12 crawlers the X-ray reads for.
Anthropic's other crawlers.
Anthropic gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]
Questions about Claude-SearchBot.
What is Claude-SearchBot?
Claude-SearchBot is one of Anthropic's three bots. Anthropic says it navigates the web to improve search result quality for users. ClaudeBot collects content that could train Anthropic's models, and Claude-User fetches pages when people ask Claude about them.
What happens if I block Claude-SearchBot?
Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
What is the Claude-SearchBot user agent string?
Anthropic doesn't publish a full one. Its page gives Claude-SearchBot as the name to match in a robots.txt User-agent line, and a list of the IP addresses its bots come from.
Does Claude-SearchBot obey robots.txt?
Anthropic says its bots honor industry standard directives in robots.txt and that it supports the Crawl-delay extension. It asks for the lines in the robots.txt of every subdomain you want to opt out.
How many sites block Claude-SearchBot?
In our study, 12 of 1,529 sites (0.8%) close their homepage to Claude-SearchBot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them Anthropic's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.
- 1Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 2IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
Keep reading
- The free AI crawler checkerTest your robots.txt against every crawler here, and fetch your homepage as the ones that answer questions.
- The free robots.txt checkerSee what your robots.txt tells each AI crawler, page by page.
- The free robots.txt generatorPick the crawlers to allow and block, and copy the file.
- Which AI crawlers to allowThe crawlers by the job each does, and what the sites AI cites allow.
- robots.txt for AI crawlersHow crawlers read the file, with rules to copy.
- Claude-UserVisits a page when someone asks Claude a question.
- ClaudeBotCollects content that could train Anthropic's models.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account