ClaudeBot, Anthropic's crawler for model training
ClaudeBot is the crawler Anthropic uses to collect web content that could contribute to training its AI models. Restricting it in robots.txt signals that your site's future materials should be left out of Anthropic's training datasets.
Checked against Anthropic's own documentation on October 13, 2026.
What ClaudeBot is for.
Anthropic says ClaudeBot helps enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.[1]
Run by
Anthropic
Its job
Model training. It collects pages that may train AI models, or decides whether pages already crawled may be used that way.
Its name in robots.txt
ClaudeBotDoes it follow robots.txt?
Yes, by its operator's account. Anthropic says its bots honor industry standard directives in robots.txt, and that it supports the non-standard Crawl-delay extension.[1]
Its IP ranges
Anthropic publishes one list of source IP addresses for its bots: an address on it indicates the crawler is coming from Anthropic. It adds that blocking those addresses may not work as an opt out, because it stops its bots from reading your robots.txt.[1] claude.com/crawling/bots.json
Crawl-delay slows it down
Anthropic's own example sets Crawl-delay: 1 under User-agent: ClaudeBot to limit how fast it crawls.[1]
Every subdomain needs its own lines
Anthropic asks you to add the lines to the robots.txt of every subdomain you want to opt out.[1]
It doesn't get past CAPTCHAs
Anthropic says its bots will not attempt to bypass CAPTCHAs on the sites they crawl.[1]
No published user agent string for ClaudeBot.
Anthropic's page gives ClaudeBot as the name to match in robots.txt and publishes no full user agent string for it.[1]
The lines that allow and block ClaudeBot.
Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to ClaudeBot now.
Allow ClaudeBot
With no robots.txt, or one that doesn't disallow it, ClaudeBot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the
User-agent: *group only when it finds none.[2]User-agent: ClaudeBot Allow: /
Block ClaudeBot
Anthropic says that when a site restricts ClaudeBot, it signals that the site's future materials should be excluded from its AI model training datasets.[1]
User-agent: ClaudeBot Disallow: /
How many cited sites block ClaudeBot.
We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.
of cited sites close their homepage to ClaudeBot in robots.txt: 40 of 1,529.
of 1,500 robots.txt files name ClaudeBot. 38 of those close the homepage to it and 420 leave it open.
Blue: ClaudeBot. Gray: the other 12 crawlers the X-ray reads for.
Anthropic's other crawlers.
Anthropic gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]
Questions about ClaudeBot.
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for model development. Anthropic says it collects web content that could potentially contribute to training its generative AI models. Claude-SearchBot and Claude-User are separate bots for search and for users' requests.
Should I block ClaudeBot?
It depends on whether you want Anthropic to train on your pages. Restricting ClaudeBot signals that your site's future materials should be excluded from Anthropic's training datasets. Claude's search and user requests run through Claude-SearchBot and Claude-User, which have their own lines in robots.txt.
What is the ClaudeBot user agent string?
Anthropic doesn't publish a full one. Its page gives ClaudeBot as the name to match in a robots.txt User-agent line, and a list of the IP addresses its bots come from.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic says it supports the non-standard Crawl-delay extension to robots.txt, and gives the example of Crawl-delay: 1 under User-agent: ClaudeBot.
How many sites block ClaudeBot?
In our study, 40 of 1,529 sites (2.6%) close their homepage to ClaudeBot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them Anthropic's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.
- 1Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 2IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
Keep reading
- The free AI crawler checkerTest your robots.txt against every crawler here, and fetch your homepage as the ones that answer questions.
- The free robots.txt checkerSee what your robots.txt tells each AI crawler, page by page.
- The free robots.txt generatorPick the crawlers to allow and block, and copy the file.
- Which AI crawlers to allowThe crawlers by the job each does, and what the sites AI cites allow.
- robots.txt for AI crawlersHow crawlers read the file, with rules to copy.
- Claude-SearchBotIndexes pages to improve Claude's search results.
- Claude-UserVisits a page when someone asks Claude a question.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account