llms.txt and Markdown for AI Crawlers: What Google Actually Said

There is a category of SEO advice that spreads because it is cheap to act on, sounds technical, and cannot be falsified quickly. The current champion is the idea that you should publish an llms.txt file and serve your pages as raw markdown so AI crawlers can read you better.

Both ideas got tested against reality this year, and the results are unusual in that the evidence comes from Google itself rather than from a study.

The llms.txt file that Google didn’t mean to publish

Google added llms.txt files to a number of its developer and documentation sites. For a day or so, the idea that the format had been blessed looked plausible.

Then it came out that the change was a CMS-wide rollout. Mueller and the people who actually operate those sites were not aware of it. Google pulled the file from the Search developer docs within about a day. Other Google sites still have theirs.

When someone asked why those files were still up — whether it meant they were used for LLM discovery — Mueller’s answer was that neither file is findable by default, because they are not at the top level of the site, and that it is safe to assume they exist “for other purposes.” He did not say what those purposes are. He was clear that discovery is not one of them.

So the honest summary of llms.txt on Google properties: an unmanaged side effect of a content management system, not a signal of adoption.

The markdown idea, and Google’s answer to it

The second idea is to serve raw markdown to AI crawlers instead of HTML. The theory is that models read text more efficiently than markup, so you should hand them the text.

Mueller’s response to that, in a Reddit thread, was a list of questions rather than an answer:

• Are you sure they can recognize markdown on a website as anything other than a text file?
• Can they parse and follow the links?
• What happens to your internal linking, header, footer, sidebar and navigation?

His follow-up on Bluesky was blunter: converting pages to markdown is a stupid idea, and — by way of analogy — LLMs can read images, so why not turn your whole site into an image?

The analogy is the useful part. Serving a different format is not the bottleneck. Reading comprehension is not the problem you are solving. What the models lack is not text; it is confidence about what is true, and serving markdown doesn’t add any.

There is a real counterexample worth knowing, because it is often used to argue the other way. Dries Buytaert made every page on his site available as markdown for AI agents, expecting a trickle, and saw hundreds of requests from ClaudeBot, GPTBot and OpenAI’s search bot within an hour. That is a real observation — and it measures crawler appetite, not retrieval outcomes. Bots asking for a file is not the same as a model citing you because of it.

Why this matters for how you spend

The markdown-and-llms.txt advice is attractive because it is a one-time technical change. Ship it, consider the AI visibility problem handled, move on. That is the same failure mode as schema markup: a controllable on-site edit standing in for the slower, less controllable work of being described by other people.

And the same research pattern keeps appearing. The Ahrefs study found schema produced no meaningful citation uplift across AI Overviews, AI Mode and ChatGPT. Mark Williams-Cook’s test found assistants happily reading an invented, invalid JSON-LD block. Google’s people keep saying SEO for AI is just SEO, and keep declining to endorse any of these new files or formats.

Three independent results, one direction: assistants are reading the open web as text, and the things they reward are the things that were always rewarded — clear descriptions, corroboration, links that resolve.

What to do instead, in the same amount of time

If you have a spare afternoon and want it to count for AI visibility, the alternatives are unglamorous and measurable:

Fix your own consistency. One description, used identically on your homepage, your about page, your social profiles and every directory listing you already have. Inconsistency across your own footprint is the most common finding in any audit we run.

Check what is indexed. It is common to find that a third of your listings have disappeared, or were never indexed. A listing nobody can read contributes nothing to a language model either.

Get onto pages that already get cited. Ask an assistant a buyer question about your category, note the sources, and work out which of them accept submissions. This is slower than adding a file and far more likely to change the answer.

Leave the formatting alone. Your HTML is fine. It has been fine for thirty years. There is no evidence that handing crawlers a different serialization of the same page changes whether you get cited.

Which leaves the placement work as the thing that does move: https://backlinksubmit.org takes a product URL, matches it to relevant directories and communities from a handpicked list, submits manually with written copy, and shows in a console which platforms have been submitted and how each is progressing.

FAQ

Should I add an llms.txt file?

There is no evidence it affects discovery or citations. Google’s own files exist because of a CMS change, not a strategy, and Google has said they are not for discovery. Adding one costs little; expecting it to do anything is the mistake.

Does serving markdown to AI crawlers help?

Google’s John Mueller listed the technical risks — link parsing, internal linking, navigation — and said plainly that converting pages to markdown is a bad idea. No major platform has endorsed it.

Why do crawlers request markdown so eagerly then?

Because crawlers request everything they can reach. Request volume measures appetite, not value. Judge the approach by whether your citations change, not by your server logs.

What actually improves AI visibility?

Being described accurately and consistently by sources other than yourself. That is the input these systems use to decide who is a credible answer, and it is the one thing no on-site file can substitute for.