GEOFast
Sign inFree X-rayRun a free X-ray

GPTBot, OpenAI's crawler for model training

GPTBot is the crawler OpenAI uses to collect content that may train its generative AI models. Disallowing it in robots.txt tells OpenAI not to train on your pages. ChatGPT search reads pages through a different crawler, OAI-SearchBot.

Check your site for GPTBot

Checked against OpenAI's own documentation on October 13, 2026.

The crawler

What GPTBot is for.

OpenAI says GPTBot is used to crawl content that may be used in training its generative AI foundation models.[1]

  • Run by

    OpenAI

  • Its job

    Model training. It collects pages that may train AI models, or decides whether pages already crawled may be used that way.

  • Its name in robots.txt

    GPTBot

  • Does it follow robots.txt?

    Yes, by its operator's account. OpenAI gives GPTBot as a robots.txt tag for managing how a site's content works with its AI.[1]

  • Its IP ranges

    OpenAI publishes GPTBot's IP addresses as a JSON file.[1] openai.com/gptbot.json

  • One crawl can serve two uses

    If a site allows both OAI-SearchBot and GPTBot, OpenAI says it may use the results of one crawl for both search and training, to avoid crawling twice.[1]

User agent

The GPTBot user agent string.

OpenAI gives this as an example and says the version number may change. When it fetches a robots.txt file it may add a robots.txt marker to the string.[1]

  • Fetching pages

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
robots.txt

The lines that allow and block GPTBot.

Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to GPTBot now.

  • Allow GPTBot

    With no robots.txt, or one that doesn't disallow it, GPTBot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the User-agent: * group only when it finds none.[2]

    User-agent: GPTBot
Allow: /

  • Block GPTBot

    OpenAI says disallowing GPTBot indicates a site's content should not be used in training its generative AI foundation models. Each setting is independent, so a site can disallow GPTBot and still allow OAI-SearchBot to appear in ChatGPT's search results.[1]

    User-agent: GPTBot
Disallow: /
Our data

How many cited sites block GPTBot.

We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.

3.1%

of cited sites close their homepage to GPTBot in robots.txt: 47 of 1,529.

491

of 1,500 robots.txt files name GPTBot. 45 of those close the homepage to it and 446 leave it open.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: GPTBot. Gray: the other 12 crawlers the X-ray reads for.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.
Same operator

OpenAI's other crawlers.

OpenAI gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]

FAQ

Questions about GPTBot.

  • Should I block GPTBot?

    It depends on whether you want OpenAI to train on your pages. Disallowing GPTBot tells OpenAI your content should not be used to train its generative AI foundation models. Allowing it lets OpenAI crawl content that may be used in that training. Neither choice changes ChatGPT search, which reads pages through OAI-SearchBot.

  • What is the GPTBot user agent string?

    OpenAI's example is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI says the version number may change.

  • Where does OpenAI publish GPTBot's IP ranges?

    At https://openai.com/gptbot.json. OAI-SearchBot and ChatGPT-User each have a list of their own.

  • Does blocking GPTBot remove my site from ChatGPT?

    Not from ChatGPT search. OpenAI says its settings are independent: OAI-SearchBot decides whether a site is shown in ChatGPT search answers, and GPTBot only covers training. If you allow both, OpenAI may use one crawl for both uses.

  • How many sites block GPTBot?

    In our study, 47 of 1,529 sites (3.1%) close their homepage to GPTBot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow

Sources

Where each fact comes from.

Each numbered source is the page a fact comes from, most of them OpenAI's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.

  1. 1OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
  2. 2IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html

Keep reading

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account