llms.txt: What It Does, What It Doesn't, and Why We Keep Ours

For anyone who has been told to add an llms.txt file, or to pay for one, this shows what the file is, what Google and the available studies say it does, and the ten-minute version worth keeping.

Pranav Unni, Founder and Lead Verifier of ThriveFinity Pranav UnniFounder and lead verifier
Published
7 minRead time
A small text file at the root of a site: a coding agent reads it, the search index and the AI answer do not.

llms.txt is a plain text file in Markdown at the root of a website that summarises what the site is and lists its important pages for language models and AI agents. It takes about ten minutes to write. On Google's own account it does nothing for AI Overviews, AI Mode or rankings, and the two largest published checks of server logs report that most such files are never fetched by any AI crawler. We publish one anyway, and this page explains why that is not a contradiction: it is a courtesy summary for the few systems that read it, not a channel, and nobody should sell it to you as one.

This page is for founders and marketing leads at B2B companies who have been told to add the file, or to pay for one. It sets out what the file is, what Google and the available studies say, what ours contains, and what to spend the time on instead.

What llms.txt Is

The format was proposed in September 2024 by Jeremy Howard of Answer.AI as a convention: a file at /llms.txt with a title, a short description, and lists of links with one-line notes, written so that a model with a small context window can understand the site without crawling all of it. A companion /llms-full.txt can carry the full text of the key pages in one file. It is a proposal, not a standard adopted by any search engine, and it asks nothing of the reader: a system can fetch it or ignore it.

The idea borrowed its shape from robots.txt, and that is where the confusion starts. robots.txt is a control: crawlers that honour it change their behaviour. llms.txt is a description: it changes nothing unless something reads it and chooses to use it.

What Google Says

Google published its first guide to optimising for generative AI features in Google Search on 15 May 2026. Search Engine Journal's report of the guide records the position plainly: site owners do not need to create machine-readable files, AI text files, markup or Markdown to appear in generative AI search, and the guide groups llms.txt with content chunking, AI-specific rewriting and special schema as tactics that are not needed for Google's features. On 16 June 2026 Search Engine Roundtable reported an update to the same guide clarifying that llms.txt files neither help nor hurt rankings and that AI files and Markdown files are not used in Google Search.

Google says llms.txt, content chunking, AI-specific rewriting, and special schema aren't needed

Search Engine Journal, On Google's generative AI optimisation guide, May 2026 · Search Engine Journal

The guide covers Google's features only. It does not speak for ChatGPT, Claude or Perplexity. And Google is not of one mind on the file: Chrome's Lighthouse tool added an experimental agentic-browsing audit that checks whether a site has an llms.txt, on the reasoning that a browsing agent without one may spend more time crawling to understand the site's structure. The two positions are about two different things. Search is about which pages get cited from an index that already exists. Agentic browsing is about an agent reading your site live, on behalf of a user, to complete a task. For citations in AI search, the file does nothing. For an agent reading your site live, it may save a few steps.

Who Reads It

The evidence on readership comes from vendors' log studies and should be read as such. Ahrefs reported in June 2026, from an analysis of about 137,000 sites, that roughly 28% had published an llms.txt and that 97% of those files were never fetched in a full month. Otterly.AI reported from a 90-day experiment on one domain that 84 of 62,100 AI bot requests were for llms.txt, about 0.1% of crawler traffic. Both are vendor studies with their own methods, so take the shape rather than the decimals: a large share of sites publish the file, and very few AI systems ask for it.

The systems that do fetch it are mostly coding agents and documentation tools that are pointed at a site by a developer, and the tools built to audit llms.txt files themselves. That matches the file's origin: it was designed for documentation sites, where a developer's assistant benefits from a curated map. A B2B marketing site is not that, and a general-purpose assistant answering a buyer's question is not that reader.

What Ours Contains and Why We Keep It

Our llms.txt is about 22 KB and our llms-full.txt about 36 KB, both live and both reachable by every AI crawler we name in robots.txt. The short file opens with a one-paragraph description of the company, then a disambiguation line naming similarly named businesses we are not, then the two services, who they are for, how we work, and a list of the pages that answer buyer questions with a one-line summary each: FAQ, pricing, terms, the comparison pages, the guides. It states the same facts as our About page, our LinkedIn page and our directory listings: name, markets, founding year, registered office.

We keep it for three reasons that have nothing to do with ranking. It costs ten minutes a month to maintain because it is generated from the same source as the sitemap. It is the one place where the company description, the disambiguation line and the page list sit together in plain text, which makes it a useful check on our own consistency. And if an agent or a tool does fetch it, the description it finds is the one we wrote. What we have not done is measure fetches of our own file: our crawler audit of 18 October 2026 checked that every named AI crawler can reach it and that it returns the right content, not whether any of them asked for it. We will pull the logs and report the count in a later post, including if the answer is none.

What to Do Instead

If the question behind "should I add llms.txt" is "how do I get described correctly in AI answers", the time goes to pages that are read. In our October panel run, the pages cited when buyers asked about us by name were the FAQ, the homepage, the terms, a comparison page and the pricing page. The unbranded buyer questions were answered from third-party pages: directories, trade press, LinkedIn, professional bodies. Neither list contains llms.txt.

  1. Write the FAQ as the questions buyers ask about you, one question per entry, answered in the first sentence, with a date on anything that can change. It was the most cited page about us.
  2. Put the same company description everywhere. Site, LinkedIn, Google Business Profile, Crunchbase, directories, Wikidata if you have an entry. An assistant describing you pulls whichever one ranks, so they must agree. This is the first step of our AI Presence service and most of it is a spreadsheet job.
  3. Check crawler access, which is the control that matters. robots.txt decides whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended can read your pages at all. A blocked crawler cannot cite you, llms.txt or not.
  4. Then, if you like, write the file. Ten minutes, the same facts, the pages that answer questions. Publish it, keep it current, expect nothing from it, and do not pay anyone for it.

Limits of This Page

  • Google's guide is cited through Search Engine Journal's and Search Engine Roundtable's reports, checked on 6 November 2026. The guide itself is at developers.google.com and can change.
  • The Ahrefs and Otterly figures are vendor studies reported by third parties and were not re-run by us. Their samples and methods are theirs.
  • Whether ChatGPT, Claude or Perplexity read llms.txt when answering is stated as "no published statement found", not as proof that they never do.
  • Our own file sizes and crawler access were checked live on 6 November 2026. Fetch counts for our file have not been measured yet. Errors we find are logged on the errata page.

Spotted something out of date or wrong on this page? Email om@thrivefinity.uk and we respond within 5 working days.

❓ Common Questions

Does llms.txt help my site appear in Google AI Overviews or AI Mode?
No, on Google's own account. Its May 2026 guide to optimising for generative AI features says site owners do not need to create machine-readable files, AI text files, markup or Markdown to appear in those features, and a June 2026 update says the file neither helps nor hurts rankings. Google may crawl the file like any other file, but gives it no special treatment.
Does ChatGPT, Claude or Perplexity read llms.txt?
None of the three has published a statement that it uses llms.txt when building a search answer, as far as we could find on 6 November 2026. Coding agents and tools that are pointed at a documentation site do fetch it. For a B2B marketing site, treat the file as a courtesy summary, not a channel.
What should an llms.txt contain?
A one-paragraph description of the company in plain words, the same facts you publish everywhere else (name, what you do, who for, markets, founding year), and a short list of the pages that answer buyer questions, each with a one-line summary. Ours also carries a disambiguation line naming the similarly named businesses we are not.
Is there any risk in publishing one?
Two small ones. A stale file can describe a company you no longer are, which is the same risk as a stale About page. And a file that lists pages you would rather not have summarised is an invitation. Keep it short, keep it current, and treat it as public.

Sources

  1. Search Engine Journal. Google's New AI Search Guide Calls AEO And GEO 'Still SEO'. May 2026, reporting Google's guide to optimising for generative AI features (15 May 2026). Checked 6 November 2026.
  2. Search Engine Roundtable. Google: LLMS.txt Files Won't Help Or Hurt Your Search Rankings. 16 June 2026. Checked 6 November 2026.
  3. Search Engine Roundtable. Google Search Team Does Not Endorse LLMs.txt Files. 2026. Checked 6 November 2026.
  4. Chrome for Developers. Lighthouse agentic browsing audit: llms.txt. As reported by Search Engine Journal and others, 2026. Checked 6 November 2026.
  5. Ahrefs. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. 15 June 2026. Vendor study, reported.
  6. Otterly.AI. llms.txt and AI Visibility: Results from the Experiment. 2026. Vendor experiment, reported.
  7. Jeremy Howard, Answer.AI. The /llms.txt file. Proposal, September 2024.
  8. ThriveFinity. Crawler access audit of 18 October 2026 and live checks of /llms.txt, /llms-full.txt and /robots.txt on 6 November 2026. Method on the methodology page.
Pranav Unni

Pranav Unni

Founder · ThriveFinity Connect on LinkedIn →

Pranav Unni is the founder and lead verifier of ThriveFinity. He reads and signs every paid deliverable personally, and writes about go-to-market decisions for established B2B companies.

Share this article

Free · a few minutes

See What AI Says About You

Run the free AI Presence Signal for a first reading of whether AI assistants name and describe your company correctly on 12 buyer questions.