Which AI Crawlers to Block When Bots Are More Than Half Your Traffic.

Bots crossed 57.5% of all HTTP requests in mid-2026, the first time machines outnumbered people on the web, and Cloudflare Radar traces the tip-over to agentic AI (TechTimes, June 5, 2026). For most sites the meaning is blunt, since more than half of what your servers answer now goes to machines, and a growing share of those machines are AI crawlers that take your content and send nothing back.

The number that matters is the ratio between what a vendor crawls and what it sends back, not the raw share of bot traffic. Some crawlers cost you bandwidth and return real visitors, while others pull thousands of pages and return nothing, and your own server logs tell you which is which far better than any industry average.

Not all AI bots do the same job

Treating every AI crawler as one thing hides the only distinction that touches your revenue. Cloudflare's breakdown of crawl purpose puts training at nearly 80% of AI crawling, with search indexing and live user fetches together under 5% (Cloudflare). Those three jobs each call for a different response.

Training bots scrape your pages to build models and hand back nothing you can measure, and ClaudeBot and GPTBot are the ones you will see most. Search-indexing bots like OAI-SearchBot and PerplexityBot fold your content into AI search products, which can surface citations and the occasional click. Live-retrieval bots, ChatGPT-User and Perplexity-User among them, hit your page because a person just asked a question your content answers, and those turn into real visits.

OpenAI runs all three, GPTBot for training, OAI-SearchBot for its index, and ChatGPT-User for live prompts, so lumping them together in your logs erases the split that decides whether a given crawl is worth serving. Sorting the crawlers by purpose sets up every decision that follows.

Your own crawl-to-referral ratio

One metric turns log data into a decision, the number of pages a vendor crawls for each visit its product sends back. SEOmator's analysis of Cloudflare Radar data lays out how wide that range runs across vendors, and the spread is stark. Anthropic's ClaudeBot runs about 23,951 pages crawled for every visit it returns, which fits a crawler built for training rather than traffic, since Anthropic has no search product to send inbound links. OpenAI's GPTBot sits near 1,276 to 1, heavy consumption with little direct return. Perplexity comes in around 111 to 1, Microsoft's Copilot near 33 to 1, Google with Gemini and AI Overviews around 5 to 1, and DuckDuckGo close to parity at 1.5 to 1.

Read those against each vendor's business model, and your own numbers will still land somewhere else, because the vertical changes everything. The same dataset shows Perplexity running 42 to 1 on finance sites and 182 to 1 on shopping, a four-fold gap driven by how people search in each. Pull the ratios from your own logs rather than borrowing an industry mean, because the mean hides the exact variation you are trying to price.

Deciding what to block

Once you have sorted crawlers by purpose and worked out their return, the blocking policy mostly writes itself. Training-only crawlers that send no referrals can go on the block list with little cost to current traffic, since blocking them gives up visits that were never going to happen. Meta-ExternalAgent is a clean example, one of the largest AI crawlers by volume with no referral mechanism attached.

Live-retrieval bots are the opposite case, and blocking them is where sites hurt themselves. ChatGPT-User and Perplexity-User reach your page because someone asked for exactly what you offer, so shutting them out removes you from the moment a buyer is deciding. Their volume is low and their value per visit is high, which is why you segment before you block rather than swinging at all bot traffic at once.

Training crawlers from vendors who also run search products are the genuinely hard call. Blocking GPTBot today saves bandwidth, and it also keeps your content out of the model behind ChatGPT's answers. ChatGPT referrals sit near 0.2% of total referral traffic for now, though that share is climbing, so the decision turns on your time horizon, weighing current server cost against future visibility inside AI answers (Cloudflare Radar, March 2026).

Why robots.txt is not enough on its own

A blocking plan that leans on robots.txt alone has a hole in it, because robots.txt is a request that well-behaved Western vendors honor and aggressive crawlers ignore. ByteDance's Bytespider is the usual example, and Cloudflare has documented AI crawlers fetching content straight past an explicit disallow. Enforcement takes a few layers working together.

1. robots.txt disallow rules, which compliant bots respect and others read as a loose suggestion.
2. Edge or WAF rules that block by verified user agent and IP, stopping non-compliant bots before they cost you bandwidth.
3. Bot verification, since a user agent is only a text field and scrapers routinely pose as Googlebot or GPTBot to slip through.

Your robots.txt and your Cloudflare or Fastly rules do different jobs, one states a preference and the other enforces it. If you operate in Indonesia, that enforcement layer also has to account for how global crawlers treat your regional content and where your real audience actually sits.

What unmanaged bots actually cost you

Leaving bot traffic unmanaged runs up two bills. The first is your analytics, since any server-side metric that does not filter bots, raw request counts, bandwidth dashboards, log-based visit counts, ends up counting machines as if they were customers. Plan capacity against unfiltered logs and you over-provision for an audience that will never buy anything.

The second bill is infrastructure. When training traffic that returns nothing makes up 80% of AI crawling, the bandwidth and compute you spend serving it is a straight subsidy to model vendors, and for a large content site that line shows up in the CDN invoice. You can size it in an afternoon by cross-referencing your logs against your bandwidth bill.

A thirty-minute monthly check

Keeping bot traffic in line does not need a standing project, just an owner and about thirty minutes a month.

1. Pull the last 30 days of server logs, filter for verified bot traffic, and group the AI agents by purpose.
2. Compare each vendor's crawl volume against referrals from its products in GA4, using chatgpt.com and perplexity.ai as the referral sources, and recompute the per-vendor ratios.
3. Move any training-only, zero-referral crawler whose activity has grown onto your edge block list.
4. Confirm your allowed retrieval bots are still verified and still being served cleanly.

Run it at the start of each month and the trend line does most of the work, because a vendor whose crawl volume climbs while its referrals stay flat has turned from a channel into a pure cost, and you catch that shift early instead of in a quarterly review.

If you want a read on which AI crawlers actually help your vertical and which are pure overhead, our AI visibility audit maps how your brand shows up across ChatGPT, Gemini, Perplexity, and Google AI Overviews, alongside the crawl economics that sit behind your server logs. And if your own logs already show crawl volume climbing with no referrals to match, we can help you sort out what to block.