What is llms.txt, and does it do anything?
llms.txt is a Markdown file at /llms.txt that says what a site is and links to its most useful pages, written for AI agents to read. Jeremy Howard proposed it in September 2024. Google Search ignores it, and no study so far has shown that it changes how often AI answers cite a site.
Checked against the studies and help pages it cites on October 13, 2026.
What llms.txt is.
One plain-text file, written in Markdown, that tells an AI agent what your site is and which pages to read first.
A proposal from 2024
Jeremy Howard published the proposal on September 3, 2024, and llmstxt.org now carries its second version. It is open for community input, and no crawler is obliged to read the file.[1, 2]
Where it lives
At the root of the site, as /llms.txt, or at any path under it, such as /docs/llms.txt. A file covers the pages under its path.[1]
What goes in it
Markdown in a set order: a top heading with the site's name, which is the only required part; a quoted line that sums the site up; any notes; then second-level headings, each over a list of links with an optional note. Links under a heading named Optional are ones an agent may skip.[1]
Markdown copies of pages
The proposal also asks sites to serve a clean Markdown version of each useful page at the same address with .md added, so the links in llms.txt lead to text an agent can read without the page's navigation and scripts.[1]
An llms.txt example.
The shape is short enough to write by hand. The llms.txt generator drafts one from your pages and checks a file you already have.
The shape every file follows
One top heading, one quoted summary, then lists of links under second-level headings. Nothing else may use a heading.[1]
# Site name > One or two sentences on what the site is. Notes an agent needs before it follows the links. ## Docs - [Page title](https://example.com/page.md): what the page covers ## Optional - [Page title](https://example.com/more.md)
A real one: ours
The top of the file GEOFast serves, shortened. The full file goes on to list our tools, guides and comparisons.
# GEOFast > GEOFast, at trygeo.fast, shows how AI assistants such as ChatGPT, Perplexity and Gemini read and recommend a website, and what to fix first. It is unrelated to other products and companies named GeoFast. ## Pages - [GEOFast](https://trygeo.fast/): the free AI visibility X-ray, how it works, its accuracy, pricing and FAQ - [How we measure AI answers](https://trygeo.fast/methodology): where the questions come from, how answers are collected and counted, and how sure each number is - [GEOFastBot](https://trygeo.fast/bot): what our crawler fetches, and how to block it
What the studies show.
Three studies looked at who has the file, who fetches it and whether it goes with being cited. None found an effect on AI answers.
of about 38,000 llms.txt files got no request from any bot or person in May 2026, in Ahrefs' study.[3]
of the requests that did arrive came from AI retrieval bots such as OAI-SearchBot and PerplexityBot.[3]
of nearly 300,000 domains had the file in SE Ranking's study, which found no link between it and how often AI answers cite a domain.[4]
of the 584,107 files Common Crawl read have the full shape the proposal asks for. A template or plugin wrote 68.27% of them.[2]
Ahrefs took every domain in its Web Analytics product that had traffic in May 2026, 137,210 in all, and counted requests for the root /llms.txt only. It says its customers are more technical than the web at large, so its 28% share of domains with a file is an upper bound, and that a fetched file isn't proof anything read it.[3]
SE Ranking compared domains with and without the file, using Spearman correlation and an XGBoost model; taking the file out of the model made it more accurate. It says its results depend on the model and data it tested.[4] Common Crawl asked 5,167,831 sampled hosts for /llms.txt in its July 2026 crawl and studied what the files hold, not what they do.[2]
Ahrefs and SE Ranking sell SEO software with AI tracking, and we sell AI tracking. We found no study that added the file to some sites and not to others and then compared the answers.
Who reads llms.txt today.
The file was written for agents that look something up while helping a person. So far the readers on record are coding tools, far more than AI search.
Google Search: no
Google says Google Search doesn't use llms.txt or other AI text files, in its results or its AI features, and that having one neither helps nor harms your visibility. It may still crawl the file as it would any other.[5]
OpenAI, Anthropic, Perplexity: not stated
Their crawler pages name robots.txt as the file their crawlers follow. None says a crawler reads llms.txt on your site.[6, 7, 8] OpenAI, Anthropic and Google's Gemini team do publish one for their own developer docs.[1]
In server logs: rarely
Among named AI tools in Ahrefs' logs, GPTBot fetched the file most and Claude Code came second, ahead of every AI search bot. No AI bot asked for the file on a site that had none.[3]
Coding agents and docs tools
The proposal says the file is used most for software documentation, where coding agents follow it to API references, and that Mintlify, GitBook, Yoast SEO and Wix generate one.[1] Chrome's Lighthouse looks for it in its agentic browsing audits and marks a missing file as not applicable, since the file is optional.[9]
llms.txt vs robots.txt.
The names look alike and the jobs don't. One file sets access; the other describes.
robots.txt sets access
It has been an IETF standard, RFC 9309, since September 2022, and tells crawlers which paths they may fetch. Its rules are a request that crawlers honor, not a lock.[10] The robots.txt guide has rules for AI crawlers.
llms.txt describes
It grants nothing and blocks nothing. Common Crawl found files that tried to set crawler rules or opt out of training there, and says those sites have not opted out: robots.txt is the file that works.[2]
sitemap.xml lists everything
A sitemap lists every page a search engine may index. llms.txt is a short, chosen list, and it can link to pages on other sites.[1]
Should you add one?
It costs little, and the proposal says a short, curated file helps coding agents and docs tools find the right page.[1] Don't expect it to change what ChatGPT or Google's AI says about you: nothing so far shows that. GEOFast serves one, and our GEO audit reports whether you have one without scoring it. To write or check yours, use the llms.txt generator.
Questions about llms.txt.
Is llms.txt a standard?
Not a formal one. It is a proposal that Jeremy Howard published at llmstxt.org in September 2024, open for community input. robots.txt, by contrast, is an IETF standard, RFC 9309.
Does ChatGPT read llms.txt?
OpenAI's crawler documentation doesn't say it does. In Ahrefs' logs, GPTBot fetched llms.txt more than any other named AI tool, yet 97% of the files in that study got no request from anyone in a month.
Does llms.txt help SEO or AI visibility?
No effect has been shown. Google says the file neither helps nor harms visibility in Google Search, and SE Ranking found no link between the file and AI citations across nearly 300,000 domains.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers which paths they may fetch. llms.txt is a short Markdown summary with links to your key pages. It grants nothing and blocks nothing, so access rules belong in robots.txt.
Where do I put llms.txt?
At the root of your site, as /llms.txt, written in Markdown. The proposal also allows one at a subpath such as /docs/llms.txt, which covers the pages under that path. Make or check an llms.txt
Where each fact comes from.
Each numbered source is the page a fact comes from: published studies and the AI companies' own help pages, read on October 13, 2026. The answers above repeat facts marked in the guide. Help pages change, so check them before you act.
- 1Jeremy Howard, llmstxt.org: The /llms.txt file, v2llmstxt.org/
- 2Common Crawl: A Content Analysis of llms.txt Files from the July 2026 Crawl Archivecommoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive
- 3Ahrefs: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Readahrefs.com/blog/llmstxt-study/
- 4SE Ranking: Does LLMs.txt impact your AI visibility and citations? No, according to researchseranking.com/blog/llms-txt/
- 5Google: Optimizing your website for generative AI features on Google Searchdevelopers.google.com/search/docs/fundamentals/ai-optimization-guide
- 6OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 7Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 8Perplexity: Perplexity Crawlersdocs.perplexity.ai/docs/resources/perplexity-crawlers
- 9Chrome for Developers: Lighthouse agentic browsing audits: llms.txtdeveloper.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
- 10IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
Keep reading
- The free llms.txt generatorBuild an llms.txt in the llmstxt.org format, check a pasted file line by line, or see if a site publishes one. Free, with the evidence on what the file does.
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI visibility tools, comparedEvery tool we have checked, by starting price.
- What is GEOGEO, generative engine optimization, in plain words: how AI answers pick their sources, what research shows helps, and how to measure your AI visibility.
- AEO vs GEO vs SEOAEO vs GEO vs SEO: where each name came from, what Google and Microsoft say, the work all three share, and the four things that change with AI answers.
- Check if ChatGPT mentions your brandCheck by hand in five steps, see why one chat can't measure it (answers change, memory, location, leading questions), and where ChatGPT and Claude do help.
- Find the competitors ChatGPT recommendsA method for finding who ChatGPT recommends instead of you: which questions to ask, how many times, how to count the names and work out your share of voice.
- Does ChatGPT give everyone the same answer?No. ChatGPT writes each answer afresh and can use your memory, location, a web search and your plan's model. OpenAI's own pages, and our repeat-run numbers.
- AI visibility metricsMention rate, first-pick rate, citation rate, share of voice and sentiment: what each one counts, how to work it out, and why each needs a range.
- Get ChatGPT to recommend your businessWhat gets a business named in ChatGPT's answers: what you control, whether you can submit your business, how long changes take, and how to measure it honestly.
- Get cited by ChatGPTHow ChatGPT search finds and picks the pages it links, from OpenAI's docs, what makes a page usable as a source, and which kinds of pages our data shows cited.
- Why ChatGPT doesn't show your websiteSix reasons ChatGPT leaves a site out, in the order to check them: robots.txt, firewalls, noindex, JavaScript, Bing's index and who the pages it reads name.
- Does ChatGPT use Google or Bing?What OpenAI documents about where ChatGPT search gets results (Bing, its own crawler and index), what is only reported about Google, and what it means for you.
- Rank in Google's AI OverviewsWhat Google says you need to show up in AI Overviews, how query fan-out and nosnippet work, and what studies show: 38% of cited pages rank in the top 10.
- Rank in Google AI ModeWhat sets Google's AI Mode apart from AI Overviews: query fan-out, follow-ups, a custom Gemini and Search Console's counts, and how often the two cite one page.
- Perplexity SEOHow Perplexity finds, reads and cites pages: its two crawlers, what its docs say and leave out, the Cloudflare dispute, studies, and what our runs show.
- Which AI crawlers to allowWhich AI crawlers feed ChatGPT, Claude, Perplexity and Google's answers, which only train models, and what the 1,529 of 1,605 cited sites we could read allow.
- robots.txt for AI crawlersCopy-ready robots.txt rules that keep AI search crawlers in and training crawlers out, how each crawler reads the file, and the mistakes that lock AI out.
- Firewalls that block AI crawlersCloudflare, Vercel, AWS WAF and others can turn AI crawlers away even when robots.txt lets them in. Where each setting lives and how to let crawlers through.
- Do AI crawlers render JavaScript?Google's and Bing's crawlers render JavaScript. OpenAI, Anthropic and Perplexity don't say, and Vercel's 2024 study found theirs don't. Our data on cited pages.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account