Here is llms.txt explained in one line: it's a proposed plain-text file, placed at yoursite.com/llms.txt, that gives AI tools a short Markdown summary of your site and links to its most useful pages. It's cheap to add and harmless, but it isn't an official standard, Google has said its search doesn't use it, and it won't make AI assistants cite you on its own.
So the honest answer for most small businesses is: it's optional. If you run documentation, a software product or a large knowledge base, it can be genuinely useful. If you run a ten-page service website, your time is better spent on the content and crawl basics covered in our guide to AEO and GEO for AI Overviews and ChatGPT. Here's how to decide.
Where did llms.txt come from?
It was proposed in 2024 by Jeremy Howard, co-founder of Answer.AI, as a way to help language models use websites at inference time. The idea is that a model answering a question shouldn't have to wade through navigation menus, cookie banners and scripts to find what a site is about.
Web pages are built for browsers. A language model working within a limited context window benefits from a clean, condensed version: what this site is, what the key pages are, and where to find the details. llms.txt is that condensed version, written in Markdown because models read Markdown well.
The proposal also suggests offering Markdown copies of important pages (for example, /pricing.md alongside /pricing). Some sites go further and publish an llms-full.txt containing their full documentation in one file.
What does an llms.txt file look like?
A heading with your name, a one-paragraph summary in a blockquote, then grouped lists of links with a short note on each. That's the whole format.
# Example Clinic > A dental clinic in Melbourne offering general, cosmetic and > emergency dentistry. Open six days a week, with same-day > emergency appointments. ## Services - [General dentistry](/general.md): check-ups, cleans, fillings - [Emergency appointments](/emergency.md): same-day slots and fees ## Practical information - [Fees and payment options](/fees.md): typical price ranges - [Location and hours](/contact.md): address, parking, opening times ## Optional - [Blog](/blog.md): oral health articles
Links can be full URLs or relative paths; this example points to Markdown copies of each page, as the proposal suggests. The "Optional" section signals content a tool can skip if it's short on space. Everything else should be the pages you'd most want an AI to read before describing your business.
Who actually reads llms.txt today?
Mainly developer tools and AI coding assistants reading documentation sites; there's little public evidence that major AI search engines rely on it. That's the key fact most "you must add llms.txt" posts leave out.
What we can say with confidence at the time of writing:
- Google Search doesn't use it. Google representatives have said publicly that llms.txt isn't something Google Search relies on, and Google's guidance for AI features points to normal crawling and indexing instead.
- No major AI search provider has published a commitment to using llms.txt as a ranking or citation signal.
- Developer-focused tools do find it useful. Many software documentation sites publish one so that coding assistants can pull accurate, up-to-date docs.
- Anyone can fetch it. If a user pastes your llms.txt link into an assistant, or an agent browses your site, a clean summary helps it get your details right.
Treat llms.txt as a courtesy file for AI tools, not a ranking lever. If that changes, adding one later takes an afternoon.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt controls access, sitemap.xml lists URLs for crawling, and llms.txt summarises content for reading. They do different jobs, and llms.txt doesn't replace either.
| File | Purpose | Format | Respected by | Can block AI crawlers? |
|---|---|---|---|---|
| robots.txt | Tells crawlers which paths they may or may not crawl | Plain-text rules | Search engines and most documented AI crawlers | Yes, by user agent |
| sitemap.xml | Lists URLs you want discovered and indexed | XML | Search engines | No |
| llms.txt | Summarises your site and points to key pages for language models | Markdown | Some AI and developer tools, voluntarily | No |
This matters because people sometimes add llms.txt thinking it will stop AI from using their content. It won't. If you want to allow or block crawlers such as GPTBot, OAI-SearchBot or PerplexityBot, that happens in robots.txt (and sometimes your CDN's bot settings). Our post on how Perplexity chooses its sources explains why blocking by accident is a common reason sites never get cited.
Should your website have one?
Add one if your site has a lot of reference content that AI tools might need to read accurately; skip it for now if your site is small and your basics aren't done yet. Use these decision rules.
Worth doing now:
- Software products with documentation, APIs or developer guides.
- SaaS businesses where users ask assistants how features work.
- Large knowledge bases, help centres or course libraries.
- Businesses whose details are frequently described wrongly by AI assistants, where a clean canonical summary may help.
Fine to skip for now:
- Small service websites with fewer than 20–30 pages.
- Sites where pages are slow, blocked to crawlers or missing basic structure. Fix those first.
- Sites without a clear owner who will keep the file accurate.
Never a substitute for:
- Crawlable, fast, well-structured pages.
- Accurate schema markup for Organization and LocalBusiness.
- A consistent presence across the web, which is what drives most of the effect described in our guide to getting recommended by ChatGPT.
How do you create one if you decide to?
Write it by hand in under an hour for a small site, or generate it from your CMS for a large one. Then keep it accurate.
- Write a two- or three-sentence summary of who you are, where you operate and what you offer. Use the same wording as your homepage and Google Business Profile.
- Pick 5–20 key pages. Services, pricing, process, contact, and any guides customers rely on.
- Group them under H2 headings and add a short description after each link.
- Put lower-priority pages under "Optional".
- Upload it to the root so it's reachable at
/llms.txt, served as plain text. - Check it in a browser and confirm every link works.
- Add it to your update checklist. When prices, services or locations change, the file changes too.
On custom-built sites, the tidiest approach is to generate the file from the same content that powers your pages, so a new service page automatically appears in it. That removes the biggest risk: a stale summary that contradicts your actual site.
What are the risks of adding llms.txt?
The risks are small but real: stale information and wasted effort. There's no known penalty for having one.
- Contradictions. If llms.txt says you're open on Sundays and your contact page says you aren't, you've created confusion rather than clarity.
- False confidence. Teams sometimes add the file and assume the AI-visibility job is done. It isn't.
- Exposing pages you didn't mean to highlight. Only list pages you're happy for anyone to read.
Frequently asked questions
Will adding llms.txt improve my Google rankings?
No. Google has said its search doesn't use llms.txt, so it has no effect on rankings or on Google's AI Overviews. Rankings still come from crawlable, helpful content and the usual search signals.
Can llms.txt stop AI companies from training on my content?
No. llms.txt is a summary for tools to read, not an access control. To limit crawling by specific AI bots, use robots.txt rules for their documented user agents, and check your hosting or CDN settings.
Is llms.txt the same as llms-full.txt?
They're related. llms.txt is a short index of key pages with brief descriptions; llms-full.txt is a variant some sites publish containing their complete documentation in a single Markdown file. Small business sites rarely need the full version.
How often should I update llms.txt?
Whenever the information it summarises changes: new services, new prices, new locations or removed pages. For most small sites that means a quick review each quarter, ideally alongside your other content updates.
Do I need a developer to add llms.txt?
Not usually. It's a text file uploaded to your site's root folder. Some hosting platforms make root-level files awkward to add, in which case a developer can set it up in a few minutes, or generate it automatically on a custom build.
Get the basics right before the extras
If you'd like an honest view of what your site needs for AI search, whether that's llms.txt, crawler access, schema or rewritten pages, we'll audit it and prioritise the fixes. See what we offer or request a fixed-price quote.