No Two AI Engines Read Your Prompt the Same Way
Type one sentence into an AI engine and it almost never searches for that sentence. It quietly rewrites your prompt into a handful of other queries you never see, runs them, reads what comes back, and hands you a single answer stitched from sources you were never shown choosing. That hidden step is where AI visibility is won or lost, and most GEO advice skips straight past it as if every engine did it the same way.
They do not. The popular engines share a family resemblance, decompose the prompt then retrieve more than once, but the moment you look at how each one actually does it, the differences are large enough to change your strategy. Google runs a branded, parallel, high-volume expansion it calls query fan-out. Microsoft and OpenAI rewrite your prompt into one or a few short keyword queries. Claude runs searches one after another, each shaped by the last. Perplexity routes by intent against an index it built itself. DeepSeek barely documents any of it. Treating these as one pipeline is the mistake, and it is an expensive one when you are optimizing an enterprise content estate against the wrong model of how the machine reads.
This is the part of AI search that has a verifiable paper trail, so the rest of this piece sticks to what the engines themselves have published, plus the one peer-reviewed study that actually measured what moves citation rates. Where the evidence is thin, we say so.
What query fan-out actually does
Start with the mechanic that has the clearest first-party confirmation, because it is also the most misunderstood.
Query fan-out takes one prompt and expands it into multiple related sub-queries that run against the web, with the results synthesized into a single answer. The crucial detail is that these sub-queries are synthetic. The model writes them, not the user, so most of them have little or no traditional search volume. You are not competing for the phrase the person typed. You are competing for a set of phrases no human ever searched, generated on the fly to interrogate the question from several angles at once.
This is not a theory. Google describes it in its own words. In the I/O 2025 announcement, Liz Reid, Google's Head of Search, wrote that "AI Mode uses our query fan-out technique, breaking down your question into subtopics and issuing a multitude of queries simultaneously on your behalf." Google's AI optimization guidegives a worked example: the prompt "how to fix a lawn that's full of weeds" fans out into "best herbicides for lawns," "remove weeds without chemicals," and "how to prevent weeds in lawn." For deeper research tasks, Google says its Deep Search "can issue hundreds of searches" for a single question.
Two properties of Google's version matter for the rest of this piece. The queries run concurrently, in parallel, not one after another. And the scale is large. No other vendor publicly claims this volume, and as you will see, most of them do something noticeably lighter.
The engines diverge the moment you look closely
Here is where the "it is all just query fan-out" framing falls apart. Lay the engines side by side using only what each company has confirmed, and you get four different shapes, not one.
| Engine | What the vendor confirms it does | Shape and scale |
|---|---|---|
| Google AI Mode and AI Overviews | "Query fan-out," breaking the question into subtopics and issuing many queries "simultaneously." Deep Search issues "hundreds." | Parallel, high volume |
| Microsoft Copilot (Bing) | "Generates a search query" of "a few words" sent to Bing. Its own examples show one prompt producing two separate queries. | Decomposition into one or a few keyword queries |
| ChatGPT (OpenAI) | Rewrites the prompt into one or more targeted queries, retrieves results, cites them. Can iterate on harder prompts. | Multi-query, can loop |
| Claude (Anthropic) | Generates a targeted query, then "multiple progressive searches, using earlier results to inform subsequent queries." Refines queries behind the scenes. | Sequential, agentic |
| Perplexity | Classifies the prompt by intent, routes simple questions through a single pass and complex ones through a decomposition pipeline, against its own index. | Intent-routed |
| DeepSeek | Reported to generate search keywords then look them up in a web index. No robust first-party English documentation. | Unclear, lightly evidenced |
| Sources: Google I/O 2025 blog and AI Optimization Guide; Microsoft Learn (Copilot web search); OpenAI Help Center and crawler docs; Anthropic web search announcement; Perplexity docs; DeepSeek coverage. Full links at the end. | ||
Microsoft is the most transparent of the non-Google group, and its account is the clearest evidence that "fan-out" is not universal. Its documentation states that Copilot "parses the user's prompt and identifies terms where information from the web would improve the quality of the response," then "generates a search query" that "consists of a few words" and sends it to Bing. In Microsoft's own example, a prompt about acquiring a company called Fabrikam produces two queries, "Fabrikam strategy" and "Fabrikam financials." That is decomposition, but it is two focused keyword queries, not hundreds of parallel synthetic ones. Microsoft even shows users the exact generated queries as "web search query citations," which is the most honest window any vendor offers into what was actually searched.
Claude's shape is different again. Anthropic's web search announcement says Claude "generates a targeted search query, retrieves relevant results, analyzes them for key information," and can also "conduct multiple progressive searches, using earlier results to inform subsequent queries." The operative word is progressive. Claude's default is sequential and reasoning-driven, search then read then decide what to search next, which is closer to iterative research than to Google's simultaneous burst. Developers cap how many searches it runs with a `max_uses` parameter.
OpenAI confirms the outcome, citations from web results surfaced by its search crawler, in its crawler documentation, and its help materials describe rewriting the prompt into targeted queries. The granular pipeline you will see in some SEO blogs, exact page counts and recursion limits, is reverse-engineered, not confirmed by OpenAI, so treat those specifics with care.
The practical takeaway is not trivia. If you optimize as though every engine runs Google-scale parallel fan-out, you over-invest in breadth of sub-topic coverage for engines that only fire two keyword queries, and you under-invest in the sequential, evidence-led reading pattern that Claude and ChatGPT actually use. Same content estate, different exposure depending on the engine.
Where the index lives decides almost everything about retrieval
Once an engine has its sub-queries, it has to retrieve. This is the second place the engines split, and the split has a direct consequence for whether your Google visibility transfers anywhere.
The deciding question is whether the engine retrieves from an existing search index or one it built itself, and whether it also fetches your live page on demand.
Google grounds its AI answers in its own Search index. Its optimization guide is explicit that AI features are "rooted in our core Search ranking and quality systems" and use retrieval-augmented generation, which Google also calls grounding, to "retrieve relevant, up-to-date web pages from our Search index." The eligibility rule is stated plainly: "a page must be indexed and eligible to be shown in Google Search with a snippet" to appear in generative AI features. There is no separate, more permissive AI crawl. If a `nosnippet` directive removes you from snippets, it removes you from AI answers too.
ChatGPT and Copilot do not have their own web index. They retrieve from Bing, then read selected pages. Microsoft's documentation describes sending the generated query to "the Bing search service" and composing the answer from what comes back, after grounding, provenance, and semantic similarity checks. Claude retrieves through a third-party search provider and can also fetch a specific URL with a separate web fetch tool added in September 2025.
Perplexity is the outlier that matters most. It runs its own crawler and, by its own account, surfaces and links sites from its own index. Its crawler documentation distinguishes PerplexityBot, which indexes content to surface it in results, from Perplexity-User, which browses live on behalf of a person. The consequence is blunt: being in Google's index does not put you in Perplexity's. It is a separate discovery surface with its own crawl, and if you have never confirmed PerplexityBot can reach your priority templates, you do not know whether you exist there at all.
What governs which sources make it into the answer
This is the question most clients actually want answered: of everything an engine retrieves, what decides which sources and which passages end up in the synthesized response. Combining the first-party statements above with the peer-reviewed research below, the decision runs through roughly six gates, in descending order of how load-bearing each one is.
| What governs selection | Why it decides | How load-bearing |
|---|---|---|
| Indexed and eligible | If the engine cannot retrieve you, not in the relevant index, blocked from crawling, or not snippet-eligible, nothing else matters. Google states this outright. | Decisive |
| Relevance to the sub-query | Because the prompt is decomposed, you are matched against a specific synthetic sub-query, not the original prompt. Narrow, well-targeted passages win. | Very high |
| Source quality and authority | Google routes everything through "core Search ranking and quality systems." Microsoft runs provenance and grounding checks. Engines pull from sources their ranking already trusts. | High |
| Freshness | Grounding exists largely to inject up-to-date information. Recency is an explicit motivation across Google, Microsoft, and Anthropic. | High, varies by query |
| Crawl and access permissions | robots.txt, snippet directives, and paywalls gate what can be read and quoted at all. | Gating |
| Extractability and attributability | A passage that cleanly answers the sub-query, can be quoted, and can be cited back is easier to use. This is where content structure starts to count. | Tiebreaker, rising |
| Sources: Google AI Optimization Guide; Microsoft Learn (Copilot web search); Anthropic web search announcement; Aggarwal et al., GEO, arXiv 2311.09735. | ||
Notice the order. The first three gates are inherited almost entirely from classic ranking and authority. The romantic idea that AI search rewards some new kind of writing, independent of whether you rank, does not survive contact with the documentation. Retrieval is still the ranking layer. What is genuinely new sits at the bottom of the list, extractability, and it is rising in importance, especially on the engines that fetch and read your live page rather than leaning on a pre-ranked index.
The crawler layer is a control panel most teams never touch
There is a practical insight buried in the retrieval question that deserves its own table, because it is where enterprise teams accidentally make themselves invisible. Different bots do different jobs, and the robots.txt control for each is separate. Blocking the training crawler does not remove you from the search experience. Blocking automated crawling does not necessarily stop a live, user-triggered fetch.
| User agent | Owner | Block it and... |
|---|---|---|
| Googlebot | You leave Google Search and its AI features. The same crawl powers both. | |
| Google-Extended | You opt out of Gemini model training only. You stay indexed and shown. | |
| OAI-SearchBot | OpenAI | You "will not be shown in ChatGPT search answers." This is the one to allow for ChatGPT visibility. |
| GPTBot | OpenAI | Your content is not used to train OpenAI foundation models. Separate from search. |
| ChatGPT-User | OpenAI | A user-initiated visit. Not automated crawling, and robots.txt "may not apply" because a person triggered it. |
| PerplexityBot | Perplexity | You are not indexed by Perplexity. The domain, headline, and a brief summary may still appear. |
| Bingbot | Microsoft | You leave Bing, which is the index grounding ChatGPT and Copilot. |
| Sources: Google crawler docs; OpenAI "Overview of OpenAI Crawlers"; Perplexity crawler docs. Quoted phrases are first-party. | ||
OpenAI now notes that it may share a single crawl between OAI-SearchBot and GPTBot to avoid duplicate fetching, so the two are linked even though their controls are separate. And the user-initiated agents, ChatGPT-User and Perplexity-User, behave like a human clicking a link, which is why a robots.txt rule that stops the automated crawler may not stop a live fetch a user triggers. If your client has reflexively blocked AI bots at the edge, this table is the conversation to have before they wonder why they vanished from ChatGPT.
The format question, where the loudest vendor disagrees with the data
Now the part everyone wants a rule for, and where the evidence genuinely pulls in two directions. Both directions are worth stating honestly, because the reconciliation is the actual insight.
Google's position is that format hacks are mostly a myth, and it says so directly. Its optimization guide tells you that you do not need to chunk content ("There's no requirement to break your content into tiny pieces"), do not need an llms.txt file or special markup, do not need special schema ("Structured data isn't required for generative AI search"), and do not need to rewrite for AI. On semantic HTML it is lukewarm: worth doing for human readability and accessibility, but "focus on human readability and don't worry about perfect code." Google's positive guidance is about substance, a unique first-hand point of view, non-commodity content, and organizing pages "by paragraphs and sections, along with headings." We argued this guide is real but incomplete in our piece on [why Google's AI guide is not the whole map](/blog/google-ai-optimization-guide-not-the-whole-map), and on the [HTML versus Markdown question](/blog/html-vs-markdown-for-ai-crawlers) specifically.
Set against that, the most rigorous study available finds that format and evidence density measurably move citation rates. The GEO paper from a Princeton-led team, presented at KDD 2024, ran a controlled experiment over a 10,000-query benchmark and found that adding citations, quotations from relevant sources, and statistics can lift a source's visibility in generative engines by up to roughly 40 percent. Large-scale industry analyses point the same way on structure: a study of tens of thousands of URLs reported by Search Engine Land found listicles, articles, and product pages drive over half of all AI citations, with query intent predicting format better than industry or model does.
These only look contradictory if you collapse two different questions. What makes a page eligible and retrievable is substance, authority, indexability, and freshness, and Google is right that format is largely irrelevant there. What makes an already-eligible passage easy to lift into an answer is clean structure and verifiable evidence, a stat, a quote, a clear source, and that is exactly what the GEO study measured. The format payoff is real, it is strongest on the engines that fetch and extract live, and it is a tiebreaker on top of retrievability, not a substitute for it. Worth noting too that format advantages decay: tracking studies have already shown listicle citations falling as engines adjust, so do not build a strategy on a single format.
What this means if you run an enterprise content program
A few implications fall straight out of the evidence, written for someone managing a real content estate rather than a single page.
The retrieval layer is the ranking layer, so audit it first. Every index-grounded engine pulls from a ranked index gated by quality systems and snippet eligibility. Before any GEO tactic, confirm your priority templates are indexable, snippet-eligible, and not quietly blocked. A `nosnippet` tag or a client-side rendering gap can remove you from AI answers without touching your blue-link rankings, which is why the diagnosis has to come before the optimization.
Optimize for the sub-question, not the head term. Because prompts are decomposed, the unit of competition is the synthetic sub-query. Cover a question's sub-themes thoroughly inside authoritative pages. Heed Google's own warning, though: do not spin up a thin separate page per fan-out variant, which trips its scaled-content-abuse policy. Depth on fewer pages beats a sprawl of near-duplicates.
Treat the engines as separate channels with separate gates. Perplexity has its own index, so Google visibility does not carry over. ChatGPT visibility runs through OAI-SearchBot and the Bing index. The crawler control matrix is individually configurable per vendor. A single robots.txt decision can open or close an entire engine.
Make your best facts extractable, then back them. Lead with original, authoritative content, then ensure the quotable parts stand alone under clear headings and carry a stat, a quote, or a named source. That is where the peer-reviewed lift comes from, and it is the layer Google has the least to say about precisely because it sits above ranking.
Separate first-party fact from tool-vendor hype. Google has publicly contradicted a large amount of popular GEO advice. When you brief a client, draw a hard line between what the engines confirm and what the optimization-tool market is selling. This entire piece is built that way on purpose.
Work with Search Agency
Most brands optimizing for AI search are building against a cartoon of how the engines work, one pipeline, one crawler, one format rule. The reality is six engines that read a prompt six different ways, retrieve from different indexes, and gate access through separate crawlers. Search Agency runs measurable GEO and AEO programs that start from what each engine can actually see and select, then fix the retrieval, structure, and evidence layer so your content is the version they cite. Explore our AI Search Optimization service when you want a strategy built on how the machines actually read, not how the tools wish they did.
