GEOFast
Sign inFree X-rayRun a free X-ray

Claude-User, the agent that fetches pages for Claude's users

Claude-User is the agent Claude uses to open a website when a person's question calls for it. Anthropic says disabling it prevents Claude from retrieving your content in response to a user's query, and that all its bots honor robots.txt.

Check your site for Claude-User

Checked against Anthropic's own documentation on October 13, 2026.

The crawler

What Claude-User is for.

Anthropic says Claude-User supports Claude AI users: when individuals ask questions to Claude, it may access websites using a Claude-User agent.[1]

  • Run by

    Anthropic

  • Its job

    A person's request. It visits a page when someone asks the assistant about it, and doesn't crawl on its own.

  • Its name in robots.txt

    Claude-User

  • Does it follow robots.txt?

    Yes, by its operator's account. Anthropic says its bots honor industry standard directives in robots.txt, and that it supports the non-standard Crawl-delay extension.[1]

  • Its IP ranges

    Anthropic publishes one list of source IP addresses for its bots: an address on it indicates the crawler is coming from Anthropic. It adds that blocking those addresses may not work as an opt out, because it stops its bots from reading your robots.txt.[1] claude.com/crawling/bots.json

  • Every subdomain needs its own lines

    Anthropic asks you to add the lines to the robots.txt of every subdomain you want to opt out.[1]

  • It doesn't get past CAPTCHAs

    Anthropic says its bots will not attempt to bypass CAPTCHAs on the sites they crawl.[1]

User agent

No published user agent string for Claude-User.

Anthropic's page gives Claude-User as the name to match in robots.txt and publishes no full user agent string for it.[1]

robots.txt

The lines that allow and block Claude-User.

Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to Claude-User now.

  • Allow Claude-User

    With no robots.txt, or one that doesn't disallow it, Claude-User is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the User-agent: * group only when it finds none.[2]

    User-agent: Claude-User
Allow: /

  • Block Claude-User

    Anthropic says disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.[1]

    User-agent: Claude-User
Disallow: /
Our data

How many cited sites block Claude-User.

We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.

1.0%

of cited sites close their homepage to Claude-User in robots.txt: 15 of 1,529.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: Claude-User. Gray: the other 12 crawlers the X-ray reads for.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.
Same operator

Anthropic's other crawlers.

Anthropic gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]

FAQ

Questions about Claude-User.

  • What is Claude-User?

    Claude-User is the agent Anthropic uses for requests that Claude's users start. Anthropic says that when individuals ask questions to Claude, it may access websites using a Claude-User agent.

  • Does Claude-User obey robots.txt?

    Anthropic says it does: its bots honor industry standard directives in robots.txt, and Claude-User lets site owners control which sites these user-initiated requests can access.

  • What happens if I block Claude-User?

    Anthropic says disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.

  • What is the Claude-User user agent string?

    Anthropic doesn't publish a full one. Its page gives Claude-User as the name to match in a robots.txt User-agent line, and a list of the IP addresses its bots come from.

  • How many sites block Claude-User?

    In our study, 15 of 1,529 sites (1.0%) close their homepage to Claude-User in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow

Sources

Where each fact comes from.

Each numbered source is the page a fact comes from, most of them Anthropic's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.

  1. 1Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  2. 2IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html

Keep reading

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account