GEOFast
Sign inFree X-rayRun a free X-ray

CCBot, Common Crawl's crawler

CCBot is the crawler of Common Crawl, a non-profit that publishes an open archive of the web. AI companies have trained models on that archive: OpenAI's GPT-3 paper lists Common Crawl as 60% of its training mix. Common Crawl says CCBot checks robots.txt before it fetches a page.

Check your site for CCBot

Checked against Common Crawl's own documentation on October 13, 2026.

The crawler

What CCBot is for.

Common Crawl describes itself as a non-profit that produces and maintains an open repository of web crawl data that anyone can access and analyze; CCBot is the crawler that gathers it. OpenAI's GPT-3 paper lists Common Crawl as 60% of its training mix.[1, 2]

  • Run by

    Common Crawl

  • Its job

    Model training. It collects pages that may train AI models, or decides whether pages already crawled may be used that way.

  • Its name in robots.txt

    CCBot

  • Does it follow robots.txt?

    Yes, by its operator's account. Common Crawl says CCBot checks robots.txt first and fetches a page only if crawling it is allowed, and that it obeys the Crawl-delay parameter.[3]

  • Its IP ranges

    Common Crawl says CCBot runs on dedicated IP address ranges with reverse DNS under crawl.commoncrawl.org, except over IPv6, and publishes the ranges as a JSON file.[1] index.commoncrawl.org/ccbot.json

  • It doesn't run JavaScript

    Common Crawl says that currently JavaScript is not executed and cookies are not used.[3]

  • Crawl-delay slows it down

    Common Crawl's example sets Crawl-delay: 2 under User-agent: CCBot to keep requests at least 2 seconds apart.[3]

User agent

The CCBot user agent string.

Common Crawl says it may increment the version number, and that it is aware of crawlers falsely identifying themselves as CCBot.[1, 3]

  • Full string

    CCBot/2.0 (https://commoncrawl.org/faq/)
robots.txt

The lines that allow and block CCBot.

Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to CCBot now.

  • Allow CCBot

    With no robots.txt, or one that doesn't disallow it, CCBot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the User-agent: * group only when it finds none.[4]

    User-agent: CCBot
Allow: /

  • Block CCBot

    Common Crawl says that with these lines its crawler will stop crawling your website, and that it keeps checking whether the robots.txt file has been updated. Its pages don't say what a block changes for the models trained on its archive.[1, 3]

    User-agent: CCBot
Disallow: /
Our data

How many cited sites block CCBot.

We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.

4.2%

of cited sites close their homepage to CCBot in robots.txt: 64 of 1,529.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: CCBot. Gray: the other 12 crawlers the X-ray reads for.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.
FAQ

Questions about CCBot.

  • What is CCBot?

    CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data anyone can access and analyze. It identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/).

  • Is CCBot an AI crawler?

    Common Crawl's pages describe an open web archive, not an AI product. The archive has been used to train models: OpenAI's GPT-3 paper lists Common Crawl as 60% of its training mix.

  • How do I verify that a request is from CCBot?

    Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS, so a reverse lookup of the address should give a name under crawl.commoncrawl.org, except over IPv6. The ranges are also published at https://index.commoncrawl.org/ccbot.json.

  • Does CCBot obey robots.txt?

    Common Crawl says CCBot checks robots.txt first and fetches a page only if crawling it is allowed. It also obeys Crawl-delay, and honors nofollow on links.

  • How many sites block CCBot?

    In our study, 64 of 1,529 sites (4.2%) close their homepage to CCBot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow

Sources

Where each fact comes from.

Each numbered source is the page a fact comes from, most of them Common Crawl's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.

  1. 1Common Crawl: CCBotcommoncrawl.org/ccbot
  2. 2OpenAI: Language Models are Few-Shot Learners (Table 2.2)arxiv.org/abs/2005.14165
  3. 3Common Crawl: FAQcommoncrawl.org/faq
  4. 4IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html

Keep reading

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account