OAI-SearchBot, the crawler behind ChatGPT search
OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT's search results. OpenAI says a site that opts out of it will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
Checked against OpenAI's own documentation on October 13, 2026.
What OAI-SearchBot is for.
OpenAI says OAI-SearchBot is for search: it is used to surface websites in search results in ChatGPT's search features.[1]
Run by
OpenAI
Its job
Search index. It crawls on its own schedule, to find pages for search results.
Its name in robots.txt
OAI-SearchBotDoes it follow robots.txt?
Yes, by its operator's account. OpenAI gives OAI-SearchBot as a robots.txt tag for managing how a site works with its AI. For search results, it says its systems can take about 24 hours to adjust to a robots.txt update.[1]
Its IP ranges
OpenAI publishes OAI-SearchBot's IP addresses as a JSON file. To help a site appear in search results, it recommends allowing requests from those ranges as well as allowing the crawler in robots.txt.[1] openai.com/searchbot.json
It is set apart from GPTBot
OpenAI says each setting is independent: a site can allow OAI-SearchBot to appear in search results and disallow GPTBot to keep its content out of training.[1]
One crawl can serve two uses
If a site allows both OAI-SearchBot and GPTBot, OpenAI says it may use the results of one crawl for both, to avoid crawling twice.[1]
The OAI-SearchBot user agent strings.
OpenAI gives these as examples and says the version number may change. When it fetches a robots.txt file it may add a robots.txt marker to the string, so the request stands out in logs that don't record paths.[1]
Fetching pages
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Fetching robots.txt
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
The lines that allow and block OAI-SearchBot.
Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to OAI-SearchBot now.
Allow OAI-SearchBot
With no robots.txt, or one that doesn't disallow it, OAI-SearchBot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the
User-agent: *group only when it finds none.[2]User-agent: OAI-SearchBot Allow: /
Block OAI-SearchBot
OpenAI says sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.[1]
User-agent: OAI-SearchBot Disallow: /
How many cited sites block OAI-SearchBot.
We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.
of cited sites close their homepage to OAI-SearchBot in robots.txt: 13 of 1,529.
of 1,500 robots.txt files name OAI-SearchBot. 10 of those close the homepage to it and 368 leave it open.
Blue: OAI-SearchBot. Gray: the other 12 crawlers the X-ray reads for.
OpenAI's other crawlers.
OpenAI gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]
Questions about OAI-SearchBot.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler for search. OpenAI uses it to surface websites in the search results of ChatGPT's search features. It is separate from GPTBot, which crawls content that may be used to train OpenAI's models.
What happens if I block OAI-SearchBot?
OpenAI says a site that is opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link. For search results, OpenAI's systems can take about 24 hours to adjust to a robots.txt update.
What is the OAI-SearchBot user agent string?
OpenAI's example is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. OpenAI says the version number may change.
Where does OpenAI publish OAI-SearchBot's IP ranges?
At https://openai.com/searchbot.json. OpenAI recommends allowing requests from those ranges, as well as allowing OAI-SearchBot in robots.txt, to help a site appear in ChatGPT's search results.
How many sites block OAI-SearchBot?
In our study, 13 of 1,529 sites (0.9%) close their homepage to OAI-SearchBot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them OpenAI's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.
- 1OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 2IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
Keep reading
- The free AI crawler checkerTest your robots.txt against every crawler here, and fetch your homepage as the ones that answer questions.
- The free robots.txt checkerSee what your robots.txt tells each AI crawler, page by page.
- The free robots.txt generatorPick the crawlers to allow and block, and copy the file.
- Which AI crawlers to allowThe crawlers by the job each does, and what the sites AI cites allow.
- robots.txt for AI crawlersHow crawlers read the file, with rules to copy.
- ChatGPT-UserVisits a page when a ChatGPT user's question calls for it.
- GPTBotCollects content that may train OpenAI's models.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account