Is your firewall blocking AI crawlers?
It can be, even when robots.txt lets them in: Cloudflare, Vercel, AWS WAF, Fastly and Akamai can each turn AI crawlers away before they reach a page. Since September 15, 2026, Cloudflare's Block option for AI training also stops Googlebot, Bingbot and Applebot, which crawl for search and training both.
Checked against each vendor's own docs on October 8, 2026.
Cloudflare: what to check.
Cloudflare sorts AI crawlers by what they do, Search, Agent and Training, and lets you allow or block each.[1] Its September 15, 2026 change made one of those choices reach further.[2]
AI bot policies: Training
Block now also stops crawlers that do search and training both, Googlebot, Bingbot and Applebot included, so your search listings go too; Block on pages with ads does the same on those pages. Disallow AI Training keeps them for search and blocks the training-only crawlers. It can't pass a no-training preference to Bing until Microsoft supports one, which Cloudflare expects in early 2027; until then, Cloudflare points to Bing's noarchive tag.[2]
AI bot policies: Search and Agent
These cover the crawlers that fetch pages for answers and for people. Keep both on Allow to stay in AI answers. Fewer than 1% of Cloudflare sites block Search bots, Cloudflare says.[1, 2]
Bot Fight Mode
WAF custom rules and Page Rules can't skip it, so a crawler it challenges stays challenged. Super Bot Fight Mode takes a custom rule with the Skip action instead.[3, 4]
Presets and migrated settings
Since September 15, 2026, a new domain starts from a preset: sites without ads allow every AI crawler, and sites with ads disallow AI training and block agents on pages with ads. Existing settings moved to the new controls: the old Block AI bots option became Disallow AI Training, with agents blocked on pages with ads.[2]
AI Crawl Control
It lets you allow or block each AI crawler by name, but Allow there doesn't override your other security rules, so check those too. If you join its pay per crawl beta, crawlers that don't pay get HTTP 402 Payment Required.[5, 6, 7]
Vercel: two rulesets to check.
AI Bots managed ruleset
Off until you turn it on. Set to Deny, it blocks every AI bot it identifies, search and user fetches as well as training. To let one through, add a WAF custom rule with the bypass action; custom rules run first.[8, 9]
Bot Protection
Off until you turn it on. It challenges traffic that isn't a browser and leaves verified bots, such as Google's crawler, alone.[9]
Other firewalls: check their bot categories.
AWS WAF Bot Control
Its CategoryAI rule blocks AI bots whether they're verified or not, unlike its other category rules. To let one through, AWS has you set that rule to Count, then add a rule after Bot Control that blocks every request with its label except the bot you want.[10, 11]
Fastly ContentGuard
It groups bots by category, AI crawlers among them, and starts in Log mode: the actions you set take effect only once you switch its protection mode to Block.[12]
Akamai
Its bot detection lets you deny the bot categories you don't want; check which of yours hold AI crawlers.[13]
Site builders block through robots.txt.
Site builders mostly don't block requests: their switches write robots.txt lines, which a crawler may or may not follow.
Squarespace
Settings, Crawlers, Block known artificial intelligence crawlers edits your robots.txt. It's off by default, and its list leaves out OAI-SearchBot, ChatGPT-User and PerplexityBot.[14]
Webflow
Two toggles in Site settings, SEO, Indexing ask search engine crawlers and AI bots to stay out, through robots.txt. Both allow by default. Its AI bots group holds ChatGPT-User and PerplexityBot, so switching it off costs you answers as well as training.[15]
WordPress.com
Settings, Reading, Prevent third-party sharing adds known AI bots to the disallow list in your robots.txt.[16]
Wix, Shopify and Ghost
No switch: you edit robots.txt yourself. Shopify calls editing
robots.txt.liquidan unsupported customization.[17, 18, 19]
Let the real crawlers through, not their impostors.
Anyone can send a request that says GPTBot. Vendors publish IP addresses or a reverse DNS check, so a firewall can let the real crawler in and keep impostors out.
OpenAI
IP lists for GPTBot, OAI-SearchBot and ChatGPT-User, as JSON.[20]
Anthropic
One IP list for its three bots. Anthropic warns that blocking by IP may not work as an opt-out; robots.txt is its way.[21]
Perplexity
IP lists for PerplexityBot and Perplexity-User, to match together with the user agent.[22]
Google
A reverse DNS lookup that ends in googlebot.com, google.com or googleusercontent.com, confirmed by a forward lookup, or its published IP lists.[23]
Microsoft
A reverse DNS lookup that ends in search.msn.com, confirmed by a forward lookup, or its IP list, which Microsoft asks you to refresh daily.[24]
Apple and Common Crawl
Reverse DNS in applebot.apple.com and crawl.commoncrawl.org, with an IP list for each.[25, 26]
Sites refuse even a polite crawler.
Refusals don't only meet AI crawlers. While we studied the sites AI answers cite, some refused our own crawler, which names itself and asked each site for one file. A refused robots.txt doesn't keep Google out on its own: Google reads any 4xx answer except 429 as having no robots.txt.[27]
cited sites answered our crawler's request for robots.txt with a refusal, such as 403 or 429.
cited pages answered our rendering browser with a challenge or a refusal.
See what your firewall tells AI crawlers.
The free AI crawler checker fetches your homepage as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot and compares their answers with a browser's. A firewall that checks crawlers by IP address may refuse our borrowed user agents and still let the real crawlers in, so the report marks that case as one to check, not as a failure.
Questions about firewalls and AI crawlers.
Does Cloudflare block AI crawlers by default?
Not on a new site without ads. Since September 15, 2026, a new domain gets a preset: sites without ads allow every AI crawler, and sites with ads disallow AI training and block agents on pages with ads. Existing settings moved to the new controls: if you had Block AI bots on, agents are now blocked on your pages with ads, so check the Agent setting.
Can blocking AI training in Cloudflare hurt my Google rankings?
Yes. Since September 15, 2026, Block under Training also blocks crawlers that do search too, Googlebot and Bingbot included. Disallow AI Training stops training and keeps search, though it can't yet tell Bing; use Bing's noarchive tag for that.
How do I let ChatGPT through my firewall?
Allow OAI-SearchBot and ChatGPT-User, matched by user agent and by OpenAI's published IP addresses, so impostors stay out. On Cloudflare, keep Search and Agent on Allow; on Vercel, add a bypass rule if the AI Bots ruleset denies them.
Does a 403 on robots.txt keep crawlers out?
Not by itself. Google reads any 4xx answer except 429 as having no robots.txt, and crawls as if nothing were blocked. Whether the firewall then refuses the pages decides what the crawler gets.
How do I know if my firewall blocks AI crawlers?
Run the free AI crawler checker. It fetches your homepage as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, compares their answers with a browser's, and names the firewall when one turns them away.
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them the vendor's own documentation, read on October 8, 2026. The answers above repeat facts marked in the guide. Vendors change these pages, so check them before you act.
- 1Cloudflare: Block AI Botsdevelopers.cloudflare.com/bots/additional-configurations/block-ai-bots/
- 2Cloudflare: Have it both ways: stay discoverable in search while disallowing AI trainingblog.cloudflare.com/accountable-mixed-use-ai-crawlers/
- 3Cloudflare: Get started with Bot Fight Modedevelopers.cloudflare.com/bots/get-started/bot-fight-mode/
- 4Cloudflare: Get started with Super Bot Fight Modedevelopers.cloudflare.com/bots/get-started/super-bot-fight-mode/
- 5Cloudflare: Manage AI crawlersdevelopers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/
- 6Cloudflare: AI Crawl Control with Cloudflare WAFdevelopers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/
- 7Cloudflare: What is Pay Per Crawl?developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/
- 8Vercel: WAF Managed Rulesetsvercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets
- 9Vercel: Bot Managementvercel.com/docs/bot-management
- 10Amazon Web Services: AWS WAF Bot Control rule groupdocs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html
- 11Amazon Web Services: Bot Control example: Allowing a specific blocked botdocs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-example-allow-blocked-bot.html
- 12Fastly: About ContentGuardfastly.com/documentation/guides/security/bot-management/contentguard/about-contentguard/
- 13Akamai: Detection methodstechdocs.akamai.com/cloud-security/docs/detection-methods
- 14Squarespace: Request that AI models exclude your sitesupport.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site
- 15Webflow: Disable search engine indexinghelp.webflow.com/hc/en-us/articles/33961368603539-Disable-search-engine-indexing
- 16WordPress.com: Make your WordPress website publicwordpress.com/support/privacy-settings/make-your-website-public/
- 17Wix: Blocking AI Crawlers from Your Sitesupport.wix.com/en/article/blocking-ai-crawlers-from-your-site
- 18Shopify: Editing robots.txt.liquidhelp.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
- 19Ghost: Modifying robots.txtghost.org/help/modifying-robots-txt/
- 20OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 21Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 22Perplexity: Perplexity Crawlersdocs.perplexity.ai/docs/resources/perplexity-crawlers
- 23Google: Verify Requests from Google Crawlers and Fetchersdevelopers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- 24Microsoft: How to verify Bingbotbing.com/webmasters/help/how-to-verify-bingbot-3905dc26
- 25Apple: About Applebotsupport.apple.com/en-us/119829
- 26Common Crawl: CCBotcommoncrawl.org/ccbot
- 27Google: How Google Interprets the robots.txt Specificationdevelopers.google.com/crawling/docs/robots-txt/robots-txt-spec
Keep reading
- The free AI crawler checkerTest 13 AI crawlers, from GPTBot to PerplexityBot, against your robots.txt, and fetch your homepage as 4 of them. Free, no sign-up.
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI visibility tools, comparedEvery tool we have checked, by starting price.
- What is GEOGEO, generative engine optimization, in plain words: how AI answers pick their sources, what research shows helps, and how to measure your AI visibility.
- AEO vs GEO vs SEOAEO vs GEO vs SEO: where each name came from, what Google and Microsoft say, the work all three share, and the four things that change with AI answers.
- Check if ChatGPT mentions your brandCheck by hand in five steps, see why one chat can't measure it (answers change, memory, location, leading questions), and where ChatGPT and Claude do help.
- Find the competitors ChatGPT recommendsA method for finding who ChatGPT recommends instead of you: which questions to ask, how many times, how to count the names and work out your share of voice.
- Does ChatGPT give everyone the same answer?No. ChatGPT writes each answer afresh and can use your memory, location, a web search and your plan's model. OpenAI's own pages, and our repeat-run numbers.
- AI visibility metricsMention rate, first-pick rate, citation rate, share of voice and sentiment: what each one counts, how to work it out, and why each needs a range.
- Get ChatGPT to recommend your businessWhat gets a business named in ChatGPT's answers: what you control, whether you can submit your business, how long changes take, and how to measure it honestly.
- Get cited by ChatGPTHow ChatGPT search finds and picks the pages it links, from OpenAI's docs, what makes a page usable as a source, and which kinds of pages our data shows cited.
- Why ChatGPT doesn't show your websiteSix reasons ChatGPT leaves a site out, in the order to check them: robots.txt, firewalls, noindex, JavaScript, Bing's index and who the pages it reads name.
- Does ChatGPT use Google or Bing?What OpenAI documents about where ChatGPT search gets results (Bing, its own crawler and index), what is only reported about Google, and what it means for you.
- Rank in Google's AI OverviewsWhat Google says you need to show up in AI Overviews, how query fan-out and nosnippet work, and what studies show: 38% of cited pages rank in the top 10.
- Rank in Google AI ModeWhat sets Google's AI Mode apart from AI Overviews: query fan-out, follow-ups, a custom Gemini and Search Console's counts, and how often the two cite one page.
- Perplexity SEOHow Perplexity finds, reads and cites pages: its two crawlers, what its docs say and leave out, the Cloudflare dispute, studies, and what our runs show.
- Which AI crawlers to allowWhich AI crawlers feed ChatGPT, Claude, Perplexity and Google's answers, which only train models, and what the 1,529 of 1,605 cited sites we could read allow.
- robots.txt for AI crawlersCopy-ready robots.txt rules that keep AI search crawlers in and training crawlers out, how each crawler reads the file, and the mistakes that lock AI out.
- What is llms.txt?llms.txt is a Markdown file that tells AI agents what a site holds. Its format, a real example, who reads it today, and what studies show about its effect.
- Do AI crawlers render JavaScript?Google's and Bing's crawlers render JavaScript. OpenAI, Anthropic and Perplexity don't say, and Vercel's 2024 study found theirs don't. Our data on cited pages.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account