AI Does Not Need a Markdown Copy of Your Website
You do not need a Markdown version of your website for AI to read it. The advice telling you to build one has the diagnosis backwards, and acting on it burns enterprise engineering time on a problem you do not actually have.
That advice is everywhere now. Export your pages to Markdown. Stand up an llms.txt at your root. Publish an llms-full.txt that flattens the whole site into one long document a model can swallow in a single pass. Every version of it rests on the same claim, that language models struggle to parse your HTML and therefore deserve a cleaner copy. The claim does not survive contact with how these systems actually read the web. HTML is not what trips them up, and a second Markdown document does nothing about the things that do. If an engine is missing your content, the format is almost never why. The content is either absent from the document the crawler downloads or buried so deep in markup noise that the signal never surfaces, and a parallel file solves neither while quietly committing you to maintaining two versions of the truth that will drift apart the moment you look away.
You do not have to take my word for any of that, or anyone else's. You can settle the question on your own site in about a minute.
Run the test that settles the argument
Open one of your most important pages. Disable JavaScript in the browser. In Chrome that is DevTools, Command Palette, "Disable JavaScript," then reload. Now look at what survives.
Can you still see the body copy, the headings, the images with their alt text, the FAQs, the internal links, the prices, the specifications, anything a customer would actually need? If the page still makes sense, you do not need a Markdown copy. The machine-readable version of your page already exists, and it is the HTML you shipped.
If the page collapses into a spinner, a logo, and an empty container, you have found the real issue, and it is not solvable with Markdown.
You can run the same check from the command line, which is closer to what most AI crawlers actually do. They request the URL, take the raw HTML the server returns, and move on. They do not wait for your framework to hydrate.
# What an AI crawler downloads: the raw HTML, no JavaScript executed. curl -sL "https://example.com/your-most-important-page" \ | sed 's/<[^>]*>//g' \ | tr -s ' \n' '\n' \ | grep -v '^$' \ | head -n 60
That strips the tags and prints the first lines of visible text from the unrendered document. If your headline, your value proposition, and your core copy show up in that output, every major crawler can read them. If the output is a handful of nav labels and a cookie banner, that is precisely what ChatGPT, Claude, and Perplexity are working with when they decide whether to cite you.
The grain of truth the trend is built on
It is worth stating the strongest version of the Markdown argument, because the trend is not pure fiction and waving it away makes you sound like you have not read the research.
In controlled extraction benchmarks, models do parse clean Markdown slightly more accurately than raw HTML. One widely circulated table-extraction comparison put Markdown around 61 percent accuracy against roughly 54 percent for the equivalent HTML, and several retrieval pipelines report double-digit gains when they ingest Markdown instead of raw page source. Those numbers are real.
They are also measuring the wrong variable for the decision in front of you. The HTML in those tests is raw production source, the kind of document that arrives wrapped in three layers of nesting, inline scripts, analytics tags, cookie logic, and a navigation tree repeated on every page. Markdown wins those benchmarks because it has a far higher signal-to-noise ratio, not because the syntax carries some special affinity for language models. Strip the noise out of the HTML and the gap closes.
This matters because it tells you what lever to pull. The benchmark is rewarding density and cleanliness, two properties you can build directly into your HTML. Converting to Markdown is one way to get a cleaner document. Writing cleaner HTML in the first place is another, and it is the one that does not leave you maintaining two copies of the truth. It is also telling that the same labs people cite as authorities reach for HTML as a first-class output format themselves. Anthropic engineers have publicly argued for generating HTML over Markdown in agent output precisely because it carries more structure, not less. The format is not the thing models struggle with.
The real bottleneck is rendering, not syntax
Here is the finding that should reorganize how you think about this. In a joint study, Vercel and MERJ analyzed crawler behavior across their network and found that none of the major AI crawlers render JavaScript. Not OpenAI's GPTBot, OAI-SearchBot, or ChatGPT-User. Not Anthropic's ClaudeBot. Not PerplexityBot, Meta's crawler, or ByteDance's. They fetch JavaScript files, in Claude's case nearly a quarter of all requests, and then they do not execute a line of it.
The mechanism is blunt. GPTBot sends an HTTP request, downloads whatever HTML the server returns, extracts the text that already exists in that response, and leaves. It does not mount your React tree. It does not wait for an API call to resolve. It does not see the content your users see, because your users have a browser running your JavaScript and the crawler does not.
This is the gap that catches enterprise teams off guard, because Googlebot does render JavaScript. It runs a headless Chrome, waits for content, and indexes the result, so a client-rendered single-page app can rank perfectly well in classic Google Search while being effectively blank to every AI crawler that visits it. The same page, two completely different readings, depending on whether the bot executes JavaScript. Gemini inherits Google's rendering through that shared infrastructure. The standalone AI crawlers do not, and they are the ones deciding whether your brand shows up in an AI answer.
A Markdown file does not touch this problem. If your core content only exists after hydration, the issue is your rendering strategy, and the fix is server-side rendering, static generation, or incremental regeneration that puts the content into the initial HTML response. As the MERJ team put it, brands have to ensure critical information is server-side rendered to stay visible across an increasingly diverse set of crawlers. That is an architecture decision, not a content-format decision.
You can spot the failure mode in seconds. A client-rendered page tends to ship a near-empty shell:
<!-- What the crawler downloads from a client-rendered SPA --> <body> <div id="root"></div> <script src="/static/js/main.8f3a2b.js"></script> </body>
Everything a human reads gets injected into that empty `root` div by JavaScript the crawler will never run. The opposite, a server-rendered page, ships the content in the response itself:
<!-- What the crawler downloads from a server-rendered page -->
<body>
<main>
<h1>Enterprise Payroll Software for Multi-Entity Companies</h1>
<p>Run payroll across 40+ countries from a single ledger...</p>
<!-- full content present, no JavaScript required -->
</main>
</body>
Same product, same framework in many cases, completely different visibility. The second page needs no Markdown twin. The first one will not be saved by one.
What llms.txt actually is, and why the engines ignore it
llms.txt deserves a direct look, because it is the specific proposal driving most of this. Jeremy Howard of Answer.AI introduced it in September 2024 as a Markdown file at the root of your site, sitemap-style, pointing language models toward curated, LLM-friendly versions of your content. The companion llms-full.txt flattens your critical pages into one long Markdown document. The intent was reasonable: context windows are finite, documentation sites are sprawling, so give models a clean map.
The problem is adoption on the consumption side, which is the only side that matters. As of mid-2026, no major AI provider has confirmed it consumes the file. Google has been explicit. Gary Illyes said at Search Central Live that Google does not support llms.txt and has no plans to, and John Mueller compared it to the long-discredited keywords meta tag, where site owners declared what their content was about instead of letting systems read the content directly. His point was the trust problem at the center of the whole idea. If an engine has to verify your llms.txt against your actual HTML anyway, because you have every incentive to describe your pages more favorably than they read, then the engine may as well parse the HTML and skip the file. Server-log analyses across large numbers of domains back this up: the major AI crawlers do not routinely request /llms.txt at all.
So you would be building and maintaining a parallel content system for an audience that, by the available evidence, is not reading it. Worse, you would be creating a second source of truth. The day your llms.txt drifts from your live pages, and it will, you are actively feeding stale or contradictory information to any system naive enough to trust it. Two documents that must agree forever is not a simplification. It is a synchronization liability dressed up as an optimization.
None of this means the people promoting llms.txt are acting in bad faith. The instinct, make your content easy for machines to consume, is correct. The implementation points at the wrong artifact.
HTML was designed to be the machine-readable layer
Strip away the trend and you are left with a format that was purpose-built for exactly this job. Semantic HTML is not a workaround for machine reading. It is the specification for it.
The difference shows up the moment you compare markup that uses the language as intended against the wrapper-heavy markup most sites actually ship. Here is the version a crawler has to guess at, where everything is a `div` and the meaning lives only in CSS class names that mean nothing to a parser:
<!-- Div soup: structurally meaningless to a machine -->
<div class="top">
<div class="big-text">How Multi-Entity Payroll Works</div>
<div class="row">
<div class="q">Can I run payroll in multiple currencies?</div>
<div class="a">Yes. The platform settles in 40+ currencies...</div>
</div>
<div class="img" style="background-image:url('diagram.png')"></div>
<div class="link" onclick="go('/pricing')">See pricing</div>
</div>
A model reading that gets a wall of generic containers. There is no signal that "How Multi-Entity Payroll Works" is the page's main heading, no indication the question and answer form a unit, no real image to attach alt text to, and a link a non-JavaScript crawler cannot follow because it is wired to an onclick handler instead of an href. Now the same content expressed semantically:
<!-- Semantic HTML: the structure is the meaning -->
<article>
<h1>How Multi-Entity Payroll Works</h1>
<section>
<h2>Can I run payroll in multiple currencies?</h2>
<p>Yes. The platform settles in 40+ currencies from a single
ledger, with automatic FX conversion at settlement.</p>
</section>
<figure>
<img src="diagram.png"
alt="Multi-entity payroll flow from local entities to a central ledger">
<figcaption>Consolidated payroll across entities.</figcaption>
</figure>
<a href="/pricing">See pricing</a>
</article>
Same words on the screen. Radically different machine readability. The heading hierarchy tells a model what the page is about and how its sections relate. The `article` and `section` elements bound self-contained units a model can extract and quote cleanly. The `img` carries alt text that survives when the image does not load. The link is a real `href` any crawler can follow. This is the clean, dense, high-signal document the Markdown benchmarks were actually rewarding, except it is your live page, not a copy of it, and it carries more structure than Markdown can express.
That last point is the one the trend keeps missing. Markdown is, in effect, HTML with fewer options. It has headings, lists, links, emphasis, and code. It cannot natively express a `figure` with a `figcaption`, an `article` boundary, a `time` element with a machine-readable datetime, accessibility roles, or structured data. When you convert a rich page to Markdown, you are not upgrading it for machines. You are throwing away the parts of the specification that exist specifically to help machines.
Structured data is the part Markdown cannot give you
The single highest-leverage machine-readable signal on a page is one that Markdown has no equivalent for: structured data. A block of JSON-LD hands the engine an explicit fact graph, who published this, when, about what entity, with what credentials, instead of forcing it to infer all of that from prose.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "How Multi-Entity Payroll Works",
"datePublished": "2026-05-22",
"dateModified": "2026-05-22",
"author": {
"@type": "Person",
"name": "Ridho",
"jobTitle": "CEO",
"worksFor": { "@type": "Organization", "name": "Search Agency" }
},
"publisher": {
"@type": "Organization",
"name": "Search Agency",
"url": "https://search.agency"
}
}
</script>
Article, FAQPage, HowTo, Organization, Person, and Product are the types that earn their keep, because they reduce extraction errors on pages that are already well structured. Schema will not rescue a thin or client-rendered page, and it is the most over-claimed lever in this entire space, but on a page that is already clean it gives engines a clean fact graph to lean on instead of guessing. There is no Markdown construct that does this. It lives in your HTML, in the document the crawlers already download, and we cover the implementation detail in the structured data work that sits inside our AI search service.
The actual move is to build the page right
If you take the energy that goes into generating and syncing parallel Markdown and redirect it into the source document, you solve the real problem once. Five things carry almost all of the weight.
1. Put the content in the initial HTML. Server-side render, statically generate, or use incremental regeneration so the body copy, headings, and key facts exist in the response before any JavaScript runs. This is the single highest-impact change for AI visibility, because it is the one that determines whether the crawler sees anything at all.
2. Use semantic elements, not div soup. Real headings in a sensible hierarchy, `article` and `section` to bound extractable units, `nav`, `main`, `figure`, and genuine `href` links. The structure is what a model reads to understand relationships.
3. Make every concrete claim extractable. Short, self-contained paragraphs under question-shaped headings, each answering its heading in the first sentence without depending on the rest of the page for context. This is the same discipline that lifts citation rates, covered in depth in our guide to structuring content for AI citations.
4. Add structured data where it maps to reality. Article, Organization, Person, Product, FAQPage, and HowTo on the pages where those types actually describe the content. Accurate schema, never decorative schema.
5. Ship it fast. AI crawlers are not patient, and the same study that found they skip JavaScript also found them spending over a third of their fetches on 404s and redirects. A slow, messy site wastes the limited attention these bots give you. Loading speed is a machine-readability feature, not just a user-experience one.
Notice what is not on that list. There is no second document. There is no format conversion. There is no separate pipeline to keep in sync. Every item improves the page your users already load and the page the crawler already downloads, which means you maintain one source of truth and it serves everyone.
What this looks like at enterprise scale
For a brochure site, this is an afternoon. For a forty-thousand-URL platform, "render your content server-side" is a program, not a checkbox, so the work has to be triaged rather than attempted everywhere at once.
Start by measuring the size of the problem instead of assuming it. Crawl your priority templates twice, once with JavaScript rendering enabled and once with it disabled, and diff the extracted text. The templates where content vanishes without JavaScript are your exposure, ranked by the traffic and revenue they touch. A product detail template that renders client-side is a five-alarm finding. A rarely-visited account settings page is not. That diff is also the cleanest way to prove the issue to engineering, because it shows the empty document in the crawler's own terms rather than as an SEO opinion.
From there it is a rendering migration sequenced by value, paired with a semantic-markup cleanup on the templates you are touching anyway. The measurement layer matters throughout, because the only way to know the work landed is to watch whether AI crawlers start reaching content they previously could not, which is exactly the kind of signal the digital measurement service is built to surface. We have run this sequence on large content estates, and the case study work shows what the recovery curve looks like when content that was trapped behind hydration becomes visible to the crawlers that were skipping it.
The teams that win the AI-visibility game are not the ones publishing a shadow copy of their site in Markdown. They are the ones whose real pages are clean, server-rendered, semantically structured, and fast, so that the document the crawler downloads is already the best version of the truth. Clean HTML is the web's native machine-readable format. It always was. The work is to write it that way, not to translate it into something with fewer options and twice the maintenance.
Most brands losing AI visibility do not have a content-format problem. They have content trapped behind client-side rendering and buried in markup that makes extraction harder than it needs to be, and no one has diagnosed which templates are actually affected. Search Agency runs measurable GEO and AEO programs that start with what the crawlers can really see, then fix the rendering and structure so your live pages, not a parallel copy, become the version AI engines read and cite. Explore our AI Search Optimization service when you want your real site to be the machine-readable one.
