The technical work that decides whether AI search can use your pages.

None of the major AI crawlers run your JavaScript, and that single fact undoes most of the advice being sold as AI search optimization right now. Vercel measured GPTBot, ClaudeBot, PerplexityBot and the rest across billions of requests and found that not one of them executes the scripts a modern page leans on, even while ChatGPT's crawler alone made 569 million requests in a single month in Vercel's data. They fetch your HTML, they fetch your script files without running them, and then they read whatever the raw HTML already contained. When your main content only appears after JavaScript executes, most of these systems never see it at all.

That reframes what AI search optimization really is. Most of it is ordinary technical SEO carrying a much shorter tolerance for mistakes, because the systems reading your site now include ones that skip steps a real browser takes for granted. Google itself says there are no additional requirements to appear in AI Overviews or AI Mode beyond being indexed and eligible for a snippet in its own documentation. The techniques have barely changed, but the cost of ignoring them has climbed sharply, because a page an answer engine cannot read is a page that no longer exists as far as that engine is concerned.

How AI systems actually reach your pages

Retrieval starts the same way classic search does, with a crawler asking for a URL and getting a response. Google is clear that its AI features draw on publicly accessible, crawlable content pulled from the index, so anything you block from crawling is anything you remove from the pool AI can answer from. The volume alone is worth taking seriously, since Vercel found AI crawlers already accounted for roughly 28 percent of Googlebot's traffic across the sites it measured.

The complication is that "the crawler" is now a dozen different user agents with different jobs. OpenAI runs GPTBot for training, OAI-SearchBot for its search product, and ChatGPT-User for live fetches when someone asks a question. Anthropic runs ClaudeBot, Perplexity runs PerplexityBot, and Google runs Google-Extended alongside the regular Googlebot. Some of these feed model training, others feed live answers, and your robots.txt can treat them separately if you want to stay out of training data while staying reachable for real-time citations.

Two things break retrieval before content ever gets a chance. The first is a robots.txt rule that blocks a bot you actually wanted, which happens more often than teams expect once someone copies a restrictive template. The second is defensive infrastructure, where an aggressive bot challenge, a rate limit, or a firewall rule silently returns a block or a soft error to exactly the crawlers you care about. Pull your server logs, filter for the AI user agents, and confirm they are getting clean 200 responses on your priority pages rather than 403s and challenges.

Why JavaScript-only pages are the biggest risk right now

This is the failure mode that catches the most sites, and it catches good sites. If your framework renders the main content client-side, a real browser fills the page in after loading the scripts, so it looks perfect to you and to anyone testing in Chrome. The AI crawlers that fetched the same URL saw an almost empty shell, because they read the HTML your server sent and stopped there. Vercel's measurement was blunt about it, finding that GPTBot, ClaudeBot, PerplexityBot, Bytespider and Meta's crawler all skip JavaScript execution entirely, even though Claude's crawler spent nearly a quarter of its requests fetching script files it never ran according to Vercel.

There is one meaningful exception. Google's Gemini rides on Googlebot's infrastructure, which does render JavaScript, and Applebot renders through a browser-based crawl as well. So a client-rendered page can still make it into Google's AI surfaces while staying invisible to ChatGPT and Perplexity. Building your visibility on the one system that happens to render is a fragile bet when the rest of the market does not.

The fix is old and well understood. Serve your primary content in the initial HTML response through server-side rendering, static generation, or prerendering for bots, so the answer is present before any script runs. You can confirm it in seconds by fetching the page with curl, or by disabling JavaScript in your browser and reloading. Whatever text survives that test is what an AI crawler gets to work with.

Getting indexed, and staying eligible

Google's rule for AI features is refreshingly short, since a page only has to be indexed and eligible to be shown with a snippet, and once it clears that bar it is already in scope for AI Overviews and AI Mode with nothing extra required. That means the things which silently pull pages out of the index are the same things that pull them out of AI answers, so an accidental noindex, a canonical tag pointing at the wrong URL, or duplicate parameterized versions splitting your signals all cost you twice now. The preview controls deserve the same scrutiny, because Google's documentation lists nosnippet, data-nosnippet, max-snippet and noindex as the levers for AI features and they cut both ways. If you use nosnippet or data-nosnippet to keep a passage out of classic snippets, you are also keeping it out of AI Overviews and AI Mode, since AI features honor the same preview settings per Google. Decide deliberately which content you want quoted, because by default a snippet-eligible page is a citable one.

What structured data really does for AI search

A lot of confident advice gets structured data exactly backwards. Google states plainly that structured data is not required for its generative AI features and there is no special schema you need to add for them in its AI optimization guidance. Schema is not a hidden switch that makes an answer engine cite you, and anyone selling it that way is overpromising.

What schema does do is disambiguate. Marking up your organization, products, and FAQs gives machines an unambiguous reading of who you are and what a page is about, which supports rich results in classic search and reinforces the entity signals that AI systems assemble when they decide who to trust on a topic. The one hard rule is that your structured data has to describe what a visitor actually sees on the page, because markup that claims prices, ratings, or answers the visible content does not contain reads as a spam signal rather than an optimization. Treat schema as clarity for the machine, add it where it maps to real on-page content, and keep your entity details consistent across your About page, your product pages, and your third-party profiles so nothing contradicts.

Writing pages an answer engine can lift cleanly

Google's AI features work through retrieval-augmented generation and a query fan-out that issues several related searches, pulls passages from the pages it retrieves, and stitches an answer together with links back out. That mechanism rewards a specific kind of writing. When the direct answer to a question sits in a clear, self-contained paragraph under a heading that names the question, the system can lift it without guesswork.

The practical moves are unglamorous and they work. Answer the question near the top of the section before you add nuance, so the useful passage is not buried under three paragraphs of preamble. Use headings that say what the section actually covers in plain language, define your terms explicitly rather than assuming context, and reach for a real table when you are comparing options across several attributes, since a comparison grid is far easier for a machine to parse than the same information smeared across prose. Every section should make sense pulled out on its own, because that is exactly how it will be used.

Keeping your facts consistent across the site

Contradictions are a trust problem, and AI systems are unusually sensitive to them because they assemble answers from several sources at once. When your pricing page, your product page, and your Google Business Profile disagree on a number, or your launch date differs from the one in your press coverage, you hand the model conflicting evidence and a reason to cite someone steadier instead. Keep the details that change, like pricing, availability, product specs, and dates, aligned everywhere they appear, and make sure your brand and entity names read the same way on your own site and on the external profiles and listings that describe you. Stale or conflicting information stopped being a cosmetic issue once machines started reading all your sources together, and it directly weakens whether an answer engine will lean on you.

llms.txt, and why it is not the shortcut people hoped for

The pitch for llms.txt is appealing. Drop a curated file at your root, point the models at your best content, and get cleaner citations. The reality is that no major system reads it yet, and Google's John Mueller said as much directly, stating that no AI system currently uses llms.txt on Bluesky in June 2025.

There is no harm in publishing one, and it may pay off later if adoption comes. Just do not let it stand in for the technical work that actually determines visibility today. A perfect llms.txt on a client-rendered site that crawlers cannot read is effort spent in the wrong place.

How to check whether any of this actually worked

Validation is the step most teams skip, and it is the one that separates a real fix from a hopeful one. Confirm the mechanics first, that your priority pages return 200 to the AI user agents in your server logs, that they are indexed in Search Console, and that the main content survives with JavaScript disabled. Those are pass-or-fail checks and they tell you the door is open.

Visibility itself needs a looser, repeated test, because a single prompt on a single day tells you almost nothing. Run a set of realistic questions your customers would actually ask across ChatGPT, Perplexity, Gemini, and Google's AI answers, then repeat them over time and watch for three things, whether you get cited, whether the citation links back to you, and whether the model describes you accurately.

Close the loop on every change you make. When you ship a fix, run the same prompts again and compare the results, so you know the outcome actually moved instead of assuming it did. That before-and-after habit is what turns AI search work into something you can report on rather than guess at.

A practical checklist

Use this as the pass you run before and after any AI search work.

• Confirm robots.txt does not block the AI crawlers you want reaching your priority content
• Check server logs to confirm those crawlers get clean 200 responses, not 403s or bot challenges
• Verify your main content is present in the raw HTML with JavaScript disabled
• Fix accidental noindex tags, wrong canonicals, and duplicate URL versions splitting your signals
• Decide which content should stay snippet-eligible, since nosnippet and data-nosnippet also remove it from AI features
• Add structured data only where it matches visible on-page content
• Keep entity and brand details consistent across your site and external profiles
• Put the direct answer near the top of each section under a plain, descriptive heading
• Use tables for genuine multi-attribute comparisons
• Keep pricing, dates, and product details aligned everywhere they appear
• Re-test real prompts across AI assistants after each change to confirm the fix landed

Where this leaves you

Strip away the hype and AI search optimization comes down to removing friction so answer engines can reach, render, read, and trust your pages. Nothing on this list is exotic, and most of it is technical SEO you were supposed to be doing anyway. The difference is that the penalty for skipping it used to be a slightly worse ranking, and now it can be complete absence from the surfaces where more of your buyers are starting their research.

If you want to know exactly where your site stands with these systems, we run a free AI Visibility Audit at search.agency/ai-search that checks the crawl, render, and eligibility issues that keep sites out of AI answers, then shows you what to fix first.