Do AI agents read Markdown? What two weeks of our own server logs say

Under the hood — the engineering behind GoodBarber, explained to the people who run apps.

To set the frame: everything below happened on our own commercial website, and our team put it in place. What we learned about the machines that read web pages applies to any content, ours and yours.

Key facts, measured on our website from September 4 to September 18, 2026:

• The training and indexing crawlers of AI companies (Amazon, Meta, OpenAI) read the Markdown pages through the .md link: about 80,000 requests in two weeks.
• Coding agents, Claude Code first, ask for Markdown with the Accept: text/markdown header and receive it: about 22,000 negotiated requests, almost all from answer engines built for agents and from Claude Code.
• ChatGPT, Claude on the web and Perplexity, when answering a person, fetched the HTML more than 13,000 times and the Markdown once.
• Google's crawler asked for zero Markdown pages.
• Markdown weighs about 80% less than the HTML page it comes from.

GoodBarber is a no-code platform for building mobile apps; goodbarber.com is its commercial website, in eleven languages, with marketing pages, a blog and a help center. When someone asks an AI assistant whether GoodBarber can build the app they have in mind, the assistant doesn't read our brochure. It fetches our pages, sometimes thousands of them a day, and squeezes an answer out of the HTML. HTML is heavy: menus, scripts, layout, cookie banners. Markdown is the text of the page and nothing else. Serving it when a machine asks means the assistant reads the real content, faster, at a fraction of the size, with a smaller chance of getting us wrong.

In August 2026, Pierre-Laurent had counted who reads our llms.txt. An llms.txt is a text file placed at the root of a website that lists its important pages for AI systems: in four months, no assistant had come for it on its own. The same logs showed something else: over the same period, assistants and their crawlers fetched our regular pages 1.6 million times. Nobody reads the map; everybody walks the streets. So the question was not "how do we get them to read the index" but "what do we hand them when they arrive".

Two doors, no duplicate content. The first is content negotiation, the standard web mechanism by which a client states in its request the format it prefers: a client that sends the header Accept: text/markdown gets the Markdown version of any public page at the same address. The second is a twin address: a client that adds .md to a page's address, /pricing.md, /blog/<article>.md, gets the same thing. This page has its twin too: add .md to its address. Every HTML page announces its twin with a standard link, the way it announces its translations. The Markdown is generated from the exact HTML the page already serves, so it can never say something the page doesn't, and it tells search engines not to index it: the HTML page stays the only one in Google.

Before switching it on, the test was whether a human visitor could notice anything: responses compared byte for byte with the twin on and off, across every kind of browser we could think of, identical, for a cost of a few microseconds per request.

We didn't invent any of this. Cloudflare converts pages to Markdown at its edge since February 2026, Zapier and Vercel serve both doors on their own sites. If you run a website and want the same, it is a day of work on the server, a conversion library, a cache, and a way of checking that nothing changed for humans. The part almost nobody does is the next one: counting.

From September 4 to September 18, 2026, two dedicated logs recorded every request for a Markdown page, on our eleven language sites, with our own traffic removed. Every "who" below is a declared name checked against the IP addresses its operator publishes, the way Pierre-Laurent did; where an operator publishes nothing, the name stays declared.

Yes, and they are the ones taking the .md link: about 80,000 requests in two weeks. Half of them come from AI companies' crawlers, the ones that build training sets and search indexes: Amazon's, Meta's, OpenAI's, all from their published IP addresses. OpenAI's crawlers find the .md version because our HTML page points to it: in 99 cases out of 100, the same crawler had fetched the HTML page in the previous ten minutes. Bing's crawler came too, though it spent most of its visit on our internal search pages, which answer any query anyone types. Google's own crawler asked for zero Markdown pages.

Yes, almost systematically. The header received about 22,000 requests, a quarter of the link's, and from a different crowd. Almost all of it comes from two search engines built for AI agents, ShapBot and ExaSearchBot, and from one coding agent: Claude Code, the tool that developers and others run on their own machine. Claude Code sent the header on virtually every one of its roughly 1,400 page requests and received Markdown 98 times out of 100, three quarters of them on our help center. That is consistent with what Checkly measured in February 2026: Claude Code, Cursor and OpenCode send the header; Codex, Gemini CLI, Copilot and Windsurf don't.

No, neither door. This is the number we would have bet wrong. In two weeks, requests made because someone asked ChatGPT a question fetched more than 10,000 of our HTML pages and a single Markdown page, three minutes after launch, most likely one of us testing. Claude on the web: about 2,700 HTML pages, zero Markdown. Perplexity: about 900, zero. They fetch the HTML page and do their own cleaning. Nothing we serve changes that.

Who reads what, and for what:

Two smaller facts. The Markdown weighs about 80% less than the HTML page it comes from, close to 85% on the help center. And before we served any Markdown, our logs already held about 900 requests for .md addresses that didn't exist, README.md, agents.md, CLAUDE.md, probed by tools that expect a website to be a code repository; one of them had guessed the exact addresses we published a month later.

Three things, from our logs rather than from anyone's slides.

Your HTML is what assistants read. ChatGPT, Claude and Perplexity fetch the page a person sees. A clean, current, well-structured page is what reaches them; a Markdown twin doesn't, at least not today. Pierre-Laurent wrote how to fix the content assistants cite when they get your business wrong; that work applies to the HTML, and it is the one that pays.

Crawlers follow the links you publish. OpenAI's crawlers took the .md link because the page offered it. If you point crawlers somewhere, they go. That is also why an llms.txt is still worth having: since our pages started linking to it, OpenAI's crawlers reach it from our own site, where they used to find it through third-party directories.

Coding agents ask for Markdown by name. If developers are part of your audience, the agents they run send Accept: text/markdown and use what they get. Our help center is where they went.

If you run a GoodBarber app, its web version is already the page an assistant can fetch and cite, and GoodBarber MCP lets an assistant help you keep that content current. The pages the assistants actually read are the ones you already publish.

Does this apply to the apps built with GoodBarber?

Not directly: this article is about our own website. What already applies to your app is the lesson: assistants read the HTML page a person sees, so keeping that content current is what matters.

Is GoodBarber's website readable by AI agents?

Yes. Since September 2026, every public page of goodbarber.com exists in Markdown, through the Accept: text/markdown header or by adding .md to its address, and every HTML page announces that version with a standard link. The site also publishes an llms.txt and an llms-full.txt, regenerated every night from the pages, and every page links to them.

Can an AI agent work with a GoodBarber app?

Yes, through GoodBarber MCP: an assistant such as Claude or ChatGPT can connect to your app to read and edit its content, publish an article or schedule a notification, with your approval. That is a different subject from this article, and it is described on the GoodBarber MCP page.

Do AI assistants read Markdown versions of web pages?

Not the consumer assistants, in our logs from September 2026. ChatGPT, Claude on the web and Perplexity fetched our HTML pages more than 13,000 times in two weeks and our Markdown pages once. Crawlers and coding agents did use the Markdown.

Does serving Markdown help with Google?

No, and it isn't meant to. Our Markdown pages tell search engines not to index them and point back to the HTML page. Google's own crawler never requested one.

Should I add an llms.txt to my site?

It costs little, and crawlers do read it when your pages link to it. In our logs it is read by crawlers, directories and SEO tools, not by the assistants that answer people's questions.