llms.txt is a plain text file in Markdown at the root of a website that summarises what the site is and lists its important pages for language models and AI agents. It takes about ten minutes to write. On Google's own account it does nothing for AI Overviews, AI Mode or rankings, and the two largest published checks of server logs report that most such files are never fetched by any AI crawler. We publish one anyway, and this page explains why that is not a contradiction: it is a courtesy summary for the few systems that read it, not a channel, and nobody should sell it to you as one.
This page is for founders and marketing leads at B2B companies who have been told to add the file, or to pay for one. It sets out what the file is, what Google and the available studies say, what ours contains, and what to spend the time on instead.
What llms.txt Is
The format was proposed in September 2024 by Jeremy Howard of Answer.AI as a convention: a file at /llms.txt with a title, a short description, and lists of links with one-line notes, written so that a model with a small context window can understand the site without crawling all of it. A companion /llms-full.txt can carry the full text of the key pages in one file. It is a proposal, not a standard adopted by any search engine, and it asks nothing of the reader: a system can fetch it or ignore it.
The idea borrowed its shape from robots.txt, and that is where the confusion starts. robots.txt is a control: crawlers that honour it change their behaviour. llms.txt is a description: it changes nothing unless something reads it and chooses to use it.
What Google Says
Google published its first guide to optimising for generative AI features in Google Search on 15 May 2026. Search Engine Journal's report of the guide records the position plainly: site owners do not need to create machine-readable files, AI text files, markup or Markdown to appear in generative AI search, and the guide groups llms.txt with content chunking, AI-specific rewriting and special schema as tactics that are not needed for Google's features. On 16 June 2026 Search Engine Roundtable reported an update to the same guide clarifying that llms.txt files neither help nor hurt rankings and that AI files and Markdown files are not used in Google Search.
Google says llms.txt, content chunking, AI-specific rewriting, and special schema aren't needed
The guide covers Google's features only. It does not speak for ChatGPT, Claude or Perplexity. And Google is not of one mind on the file: Chrome's Lighthouse tool added an experimental agentic-browsing audit that checks whether a site has an llms.txt, on the reasoning that a browsing agent without one may spend more time crawling to understand the site's structure. The two positions are about two different things. Search is about which pages get cited from an index that already exists. Agentic browsing is about an agent reading your site live, on behalf of a user, to complete a task. For citations in AI search, the file does nothing. For an agent reading your site live, it may save a few steps.
Who Reads It
The evidence on readership comes from vendors' log studies and should be read as such. Ahrefs reported in June 2026, from an analysis of about 137,000 sites, that roughly 28% had published an llms.txt and that 97% of those files were never fetched in a full month. Otterly.AI reported from a 90-day experiment on one domain that 84 of 62,100 AI bot requests were for llms.txt, about 0.1% of crawler traffic. Both are vendor studies with their own methods, so take the shape rather than the decimals: a large share of sites publish the file, and very few AI systems ask for it.
The systems that do fetch it are mostly coding agents and documentation tools that are pointed at a site by a developer, and the tools built to audit llms.txt files themselves. That matches the file's origin: it was designed for documentation sites, where a developer's assistant benefits from a curated map. A B2B marketing site is not that, and a general-purpose assistant answering a buyer's question is not that reader.
What Ours Contains and Why We Keep It
Our llms.txt is about 22 KB and our llms-full.txt about 36 KB, both live and both reachable by every AI crawler we name in robots.txt. The short file opens with a one-paragraph description of the company, then a disambiguation line naming similarly named businesses we are not, then the two services, who they are for, how we work, and a list of the pages that answer buyer questions with a one-line summary each: FAQ, pricing, terms, the comparison pages, the guides. It states the same facts as our About page, our LinkedIn page and our directory listings: name, markets, founding year, registered office.
We keep it for three reasons that have nothing to do with ranking. It costs ten minutes a month to maintain because it is generated from the same source as the sitemap. It is the one place where the company description, the disambiguation line and the page list sit together in plain text, which makes it a useful check on our own consistency. And if an agent or a tool does fetch it, the description it finds is the one we wrote. What we have not done is measure fetches of our own file: our crawler audit of 18 October 2026 checked that every named AI crawler can reach it and that it returns the right content, not whether any of them asked for it. We will pull the logs and report the count in a later post, including if the answer is none.
What to Do Instead
If the question behind "should I add llms.txt" is "how do I get described correctly in AI answers", the time goes to pages that are read. In our October panel run, the pages cited when buyers asked about us by name were the FAQ, the homepage, the terms, a comparison page and the pricing page. The unbranded buyer questions were answered from third-party pages: directories, trade press, LinkedIn, professional bodies. Neither list contains llms.txt.
- Write the FAQ as the questions buyers ask about you, one question per entry, answered in the first sentence, with a date on anything that can change. It was the most cited page about us.
- Put the same company description everywhere. Site, LinkedIn, Google Business Profile, Crunchbase, directories, Wikidata if you have an entry. An assistant describing you pulls whichever one ranks, so they must agree. This is the first step of our AI Presence service and most of it is a spreadsheet job.
- Check crawler access, which is the control that matters. robots.txt decides whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended can read your pages at all. A blocked crawler cannot cite you, llms.txt or not.
- Then, if you like, write the file. Ten minutes, the same facts, the pages that answer questions. Publish it, keep it current, expect nothing from it, and do not pay anyone for it.
Limits of This Page
- Google's guide is cited through Search Engine Journal's and Search Engine Roundtable's reports, checked on 6 November 2026. The guide itself is at developers.google.com and can change.
- The Ahrefs and Otterly figures are vendor studies reported by third parties and were not re-run by us. Their samples and methods are theirs.
- Whether ChatGPT, Claude or Perplexity read llms.txt when answering is stated as "no published statement found", not as proof that they never do.
- Our own file sizes and crawler access were checked live on 6 November 2026. Fetch counts for our file have not been measured yet. Errors we find are logged on the errata page.
Spotted something out of date or wrong on this page? Email om@thrivefinity.uk and we respond within 5 working days.
❓ Common Questions
Does llms.txt help my site appear in Google AI Overviews or AI Mode?
Does ChatGPT, Claude or Perplexity read llms.txt?
What should an llms.txt contain?
Is there any risk in publishing one?
Sources
- Search Engine Journal. Google's New AI Search Guide Calls AEO And GEO 'Still SEO'. May 2026, reporting Google's guide to optimising for generative AI features (15 May 2026). Checked 6 November 2026.
- Search Engine Roundtable. Google: LLMS.txt Files Won't Help Or Hurt Your Search Rankings. 16 June 2026. Checked 6 November 2026.
- Search Engine Roundtable. Google Search Team Does Not Endorse LLMs.txt Files. 2026. Checked 6 November 2026.
- Chrome for Developers. Lighthouse agentic browsing audit: llms.txt. As reported by Search Engine Journal and others, 2026. Checked 6 November 2026.
- Ahrefs. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. 15 June 2026. Vendor study, reported.
- Otterly.AI. llms.txt and AI Visibility: Results from the Experiment. 2026. Vendor experiment, reported.
- Jeremy Howard, Answer.AI. The /llms.txt file. Proposal, September 2024.
- ThriveFinity. Crawler access audit of 18 October 2026 and live checks of /llms.txt, /llms-full.txt and /robots.txt on 6 November 2026. Method on the methodology page.