llms.txt: what it is, who reads it and whether you need one

llms.txt is a Markdown file at the root of a website that gives AI systems a short summary of the site and links to its most important pages. It can help AI agents and developer tools understand a site quickly, but it does not improve rankings, and Google Search does not use it.

That makes it a cheap, low-risk addition, not a strategy. This guide explains what the file is, who actually reads it and when it is worth adding.

What llms.txt is

llms.txt is a proposed convention, not an official standard. It was proposed by Jeremy Howard of Answer.AI in 2024 as a way to give language models a clean, curated overview of a site instead of making them parse full HTML pages (llmstxt.org   ).

The file lives at /llms.txt and is written in Markdown: a title, a short summary, and sections of links with one line of context each. A minimal example:

# Example Store

> Outdoor clothing and gear, shipped from Germany to the EU and the US.

## Key pages

- [Shipping and returns](https://example.com/shipping): Delivery times, costs and the 30 day return policy
- [Size guide](https://example.com/size-guide): Measurements for all jackets and trousers
- [Contact](https://example.com/contact): Support hours and how to reach us

## Collections

- [Rain jackets](https://example.com/collections/rain-jackets): Waterproof jackets for hiking and city use

Some documentation platforms also publish an llms-full.txt with the full text of every page in one file. That is a convention used mainly for developer documentation, not something most business sites need.

Who actually reads llms.txt

The file's reputation is bigger than its use. Here is what each kind of reader does with it.

Why Lighthouse checks for it

Chrome Lighthouse now includes an experimental Agentic Browsing audit for llms.txt, even though Google Search does not use the file. The two positions do not conflict: Lighthouse treats llms.txt as a convenience for AI agents browsing a site, not as a search signal. Chrome's documentation says that without the file, agents may spend more time crawling a site to understand its high-level structure and primary content (Chrome for Developers   ).

The audit is lenient. If the file doesn't exist, it is marked as not applicable, because the file is optional. It only flags the page if the server returns an error when the file is requested (Chrome for Developers   ).

llms.txt vs robots.txt and sitemap.xml

llms.txt is often confused with robots.txt, but the two do completely different jobs. robots.txt decides which bots may crawl your site. llms.txt only describes it.

If your goal is visibility in AI search, robots.txt matters far more. A blocked AI search bot can't read your pages at all, no matter what llms.txt says.

Should you add an llms.txt file?

Add one if it takes little effort, and don't expect it to change your visibility. It is worth more on some sites than others.

Worth adding when:

• You publish documentation, APIs or developer resources that people use with AI coding tools.
• You want AI agents to find your key pages, such as pricing, policies and support, without crawling the whole site.
• Your platform generates it for you. Shopify now serves agent-discovery files automatically on every store: agents.md as the main file, plus llms.txt and llms-full.txt for older AI crawlers (Shopify Help Center   ).

Not a priority when:

• AI crawlers are blocked in robots.txt or by your CDN.
• Your content only appears after JavaScript runs.
• Your structured data is missing or broken.

Fix those first. They have a much more direct effect on whether AI systems can access and understand your site. llms.txt is secondary to those fundamentals.

How to create an llms.txt file

1. Pick the pages that matter. Usually ten to thirty: main services or collections, pricing, shipping and returns, support, and your most useful guides. It is a summary, not a sitemap.
2. Write one line of context per link. Say what the page answers, not just its title.
3. Save it as Markdown at the root, so it loads at https://yourdomain.com/llms.txt with a 200 status.
4. Keep it current. A file that links to deleted or redirected pages sends agents to dead ends. Update it when key pages change.
5. Check it once. Open the URL in a browser, and run Lighthouse to confirm the Agentic Browsing audit doesn't flag it.

On most content management systems, a plugin or a static file at the root does the job. On Shopify, customize the generated files through theme templates instead of replacing them.

When to get help

llms.txt is the easy part of being ready for AI search. The parts that decide whether AI systems can read and cite you are AI crawler access, content in the HTML and valid structured data. That is the technical work Developtify's AI search optimization service covers. For the bigger picture, read our GEO vs SEO guide.

Frequently asked questions

Is llms.txt actually used?

Partly. Google Search doesn't use it, and server log studies show the major AI search crawlers rarely request it. AI agents and developer tools can use it as a site summary, and Chrome Lighthouse checks whether it exists.

Is llms.txt mandatory?

No. It is an optional, proposed convention. No search engine or AI system requires it, and Lighthouse marks a missing file as not applicable rather than as a failure.

Does llms.txt improve rankings or AI citations?

There is no evidence that it does. Google says it doesn't use the file for Search, and AI search crawlers rarely read it. Crawl access, content in the HTML and structured data matter far more.

What is llms-full.txt?

A companion file some documentation platforms publish, containing the full text of every page in one Markdown file. It is mainly useful for developer documentation used with AI coding tools.

Can llms.txt block AI bots?

No. llms.txt only describes your site. To allow or block AI crawlers, use robots.txt and your CDN or firewall rules.