GEOFast
Sign inFree X-rayRun a free X-ray

Bingbot, the crawler behind Bing and Copilot

Bingbot is Microsoft's standard crawler. Microsoft says Bing and Copilot search experiences rely on the same crawling, indexing and ranking foundation, so the robots.txt lines for Bingbot decide whether both can crawl your pages.

Check your site for Bingbot

Checked against Microsoft's own documentation on October 13, 2026.

The crawler

What Bingbot is for.

Microsoft calls Bingbot its standard crawler, handling most of its crawling needs each day. Its guidelines say Bing and Copilot search experiences rely on the same core crawling, indexing and ranking foundation as traditional search.[1, 2]

  • Run by

    Microsoft

  • Its job

    Search index. It crawls on its own schedule, to find pages for search results.

  • Its name in robots.txt

    bingbot Microsoft writes the name in lower case in its user agent strings. Crawlers match the User-agent line without regard to case, so Bingbot works as well.[1, 3]

  • Does it follow robots.txt?

    Yes, by its operator's account. Microsoft says robots.txt files can be configured to tell Bing crawlers how to interact with your website, and that robots.txt controls crawl access, not indexing.[1, 2]

  • Its IP ranges

    Microsoft publishes a list of Bingbot IP addresses and asks you to refresh it daily, since it can change at any time. It also verifies by DNS: a reverse lookup of the address should give a name ending in search.msn.com, and a forward lookup of that name should give the same address.[4] bing.com/toolbox/bingbot.json

  • robots.txt doesn't remove a page

    Microsoft says to use NOINDEX when a URL should not appear in Bing search, Copilot experiences or grounding API results.[2]

User agent

The Bingbot user agent strings.

W.X.Y.Z stands for the version of Microsoft Edge that Bingbot renders with, which Microsoft keeps at the most recent stable release. Microsoft warns that user agent strings are easy to spoof.[1]

  • Desktop

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

  • Mobile

    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

  • The short form

    Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
robots.txt

The lines that allow and block Bingbot.

Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to Bingbot now.

  • Allow Bingbot

    With no robots.txt, or one that doesn't disallow it, Bingbot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the User-agent: * group only when it finds none.[3]

    User-agent: bingbot
Allow: /

  • Block Bingbot

    Bing can no longer crawl the pages, for search or for Copilot. Microsoft's guidelines list blocking Bingbot in robots.txt among the things to avoid. To keep pages in Bing and limit Copilot alone, they name NOARCHIVE, which prevents content from being used in Copilot responses and grounding results, and NOCACHE, which limits Copilot to the URL, title and snippet.[2]

    User-agent: bingbot
Disallow: /
Our data

How many cited sites block Bingbot.

We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.

0.1%

of cited sites close their homepage to Bingbot in robots.txt: 1 of 1,529.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: Bingbot. Gray: the other 12 crawlers the X-ray reads for.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.
FAQ

Questions about Bingbot.

  • Does Bingbot crawl for Copilot?

    Yes. Microsoft's guidelines say Bing and Copilot search experiences rely on the same core crawling, indexing and ranking foundation as traditional search, so there is no separate Copilot crawler to allow or block.

  • What is the Bingbot user agent string?

    Microsoft lists several. The desktop one is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36, where W.X.Y.Z is the Microsoft Edge version in use. The short form is Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm).

  • How do I verify that a request is from Bingbot?

    Microsoft gives two ways. Run a reverse DNS lookup on the IP address, check that the name ends in search.msn.com, and confirm with a forward lookup that the name returns the same address. Or match the address against the list at https://www.bing.com/toolbox/bingbot.json, refreshed daily.

  • Can I keep my pages out of Copilot and stay in Bing?

    Microsoft's guidelines name two tags for that. NOARCHIVE prevents content from being used in Copilot responses and grounding results, and NOCACHE limits Copilot to the URL, title and snippet. Blocking Bingbot in robots.txt stops crawling for both Bing and Copilot.

  • How many sites block Bingbot?

    In our study, 1 of 1,529 sites (0.1%) close their homepage to Bingbot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow

Sources

Where each fact comes from.

Each numbered source is the page a fact comes from, most of them Microsoft's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.

  1. 1Microsoft: Which Crawlers Does Bing Use?bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
  2. 2Microsoft: Bing Webmaster Guidelinesbing.com/webmasters/help/webmaster-guidelines-30fba23a
  3. 3IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
  4. 4Microsoft: How to verify Bingbotbing.com/webmasters/help/how-to-verify-bingbot-3905dc26

Keep reading

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account