# How much do AI crawlers crawl compared to what they send back?

> Cloudflare Radar’s 17 November 2026 snapshot records 3,351.73 Perplexity crawls, 2,073.96 OpenAI crawls and 456.23 Anthropic crawls per observed referral over the last seven days.

Source: https://llmsearchnews.com/ai-crawler-traffic-2026 · Publisher: LLM Search News · Published: 2026-11-17 · Updated: 2026-11-17

## TL;DR
- Cloudflare Radar’s 17 November 2026 snapshot records 3,351.73 Perplexity crawls, 2,073.96 OpenAI crawls and 456.23 Anthropic crawls per observed referral over the last seven days.[7][1]
- The OpenAI and Anthropic figures are platform-level ratios, not GPTBot-only or ClaudeBot-only measurements.[3]
- Fastly measured approximately 30% AI-request growth from January through May 2026, around 6.5 times the growth rate of human traffic.[11]
- Cloudflare Pay Per Crawl remains documented as a closed-beta payment mechanism, with no cited market-wide transaction or payout totals.[17][15]
- Cloudflare’s November 2026 access-policy documentation lists Allow as the new-domain default for Search, Agents and Training, so older default-blocking headlines need a date and qualification.[26][13]

As of **17 November 2026**, AI platforms still crawl substantially more HTML than the measurable referrals they send to publishers, but the gap varies by platform and observation window: Cloudflare Radar’s latest seven-day snapshot records about **2,073.96 OpenAI crawls per observed referral, 456.23 Anthropic crawls, and 3,351.73 Perplexity crawls**.[1][2] These are platform-level figures, not ratios for GPTBot or ClaudeBot alone, and Cloudflare warns that missing referral headers—particularly from native apps—can inflate the measured gap.[1]
## What happened?

Cloudflare Radar’s 17 November 2026 snapshot shows a large but uneven difference between HTML crawling and observed referrals. Perplexity recorded the highest current platform-level ratio at **3,351.73 crawls per observed referral**, followed by OpenAI at **2,073.96** and Anthropic at **456.23**.[3][1]

The measurement covers the dashboard’s **last seven days**, rather than the full month of November.[1] Cloudflare’s calculation counts HTML crawl requests from associated crawler user agents and compares them with HTML requests containing a platform-associated `Referer` hostname.[4]

That methodology means the figures should be described as **crawls per observed referral**, not crawls per visitor, customer, conversion or unit of publisher revenue.[1] They also do not establish how many requests came specifically from GPTBot or ClaudeBot, because the platform aggregates multiple bots under each operator.[4][1]

Cloudflare’s rendered bot-traffic chart separately displays **ClaudeBot at 12.1%** and **GPTBot at 8.1%**, but the accessible chart text does not define whether those percentages use all HTTP requests or a crawler-only denominator.[5][6] They should therefore be reported as the chart’s displayed percentages, not as shares of the entire web.[7]

## Why does the crawl-to-referral gap matter?

The gap matters because crawling and clicking produce different outcomes for publishers. A crawler can retrieve a page for training, search indexing, answer generation or an agent task without sending a person through to the publisher’s site.[8][9]

The current measurements do not show that every crawl is intended for model training. Cloudflare reported that training represented **52% of crawler requests in June 2026**, compared with **22% in spring 2025**, while its classification also distinguishes search, agent and other purposes.[8]

Fastly’s 2026 data shows that AI traffic is growing as an infrastructure workload as well as a content-distribution issue. In a fixed customer cohort covering **January 1 through May 31, 2026**, AI requests grew by approximately **30%**, around **6.5 times faster than human traffic**.[10][11]

Fastly also reported that AI requests were more likely than human requests to reach origin infrastructure: less than **9% of human requests** reached origin in May 2026, compared with more than **51% of AI requests**.[11] That difference affects caching, bandwidth, origin capacity and bot-management decisions, although the cited analysis does not provide a dollar cost per request.[8]

The economic question is separate from the technical one. A 2026 academic study using U.S. desktop clickstream data from **October 2024 through July 2025** found a clean outbound click in **5.2% of ChatGPT conversation sessions**, compared with **31.1% of Google queries**.[2] Those are user-behaviour measurements, not crawler-to-referral ratios, and neither result measures publisher revenue.[12][1]

## What do the latest measurements show?

Cloudflare’s current platform-level values are:

| Crawler or platform | Operator | Share or ratio | Source | Date |
|---|---|---:|---|---|
| GPTBot and related OpenAI bots | OpenAI | 2,073.96 crawls per observed referral | Cloudflare Radar | 17 November 2026[1] |
| ClaudeBot and related Anthropic bots | Anthropic | 456.23 crawls per observed referral | Cloudflare Radar | 17 November 2026[1] |
| Perplexity crawler group | Perplexity | 3,351.73 crawls per observed referral | Cloudflare Radar | 17 November 2026[12][5] |
| ClaudeBot | Anthropic | 12.1% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |
| GPTBot | OpenAI | 8.1% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |
| Googlebot | Google | 25.5% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |

Cloudflare’s current ratio differs sharply from its historical 2025 figures. In the monthly comparison for **January and July 2025**, Anthropic fell from **286,930.1:1** to **38,065.7:1**, OpenAI moved from **1,217.4:1** to **1,091.4:1**, and Perplexity rose from **54.6:1** to **194.8:1**.[1]

Those monthly values cannot be treated as a direct continuation of the November 2026 rolling-week snapshot. The observation windows, measurement periods and platform activity changed between the datasets.[4][14]

Cloudflare’s 2025 historical analysis also recorded different movements among individual bots. Between May 2024 and May 2025, GPTBot requests grew by **305%**, while ClaudeBot requests fell by **46%** in the fixed-customer comparison.[13][1]

The data therefore does not support one universal ranking such as “the worst crawler” or a permanent GPTBot crawl ratio. The answer depends on the operator, included user agents, time window, referral definition and traffic classification.[2]

## What is confirmed—and what is not?

### Confirmed

- Cloudflare’s 17 November 2026 Radar snapshot reports platform-level crawl-to-observed-referral ratios of **3,351.73:1 for Perplexity, 2,073.96:1 for OpenAI and 456.23:1 for Anthropic**.[1]
- The snapshot covers the dashboard’s **last seven days**, not a full-November average.[7]
- OpenAI and Anthropic platform totals aggregate multiple user agents, so the ratios cannot be assigned to GPTBot or ClaudeBot alone.[4]
- Fastly measured approximately **30% AI-request growth** from January through May 2026, with AI traffic growing about **6.5 times faster than human traffic**.[11][10]
- Cloudflare’s Pay Per Crawl documentation still describes the product as **closed beta** in documentation updated **28 July 2026**.[15]
- Cloudflare’s access-policy documentation updated **2 November 2026** lists **Allow** as the new-domain default for Search, Agents and Training.[16][13]

### Not confirmed by the cited evidence

- Absolute November 2026 request totals for GPTBot or ClaudeBot are not published in the cited Cloudflare dashboard responses.[1]
- The cited sources do not publish a universal AI-crawler-to-revenue ratio.[17]
- Pay Per Crawl’s cited documentation does not publish aggregate paid-request volume, publisher payouts or the number of paying AI operators.[8][1]
- The cited material does not provide a general-availability date for Pay Per Crawl.[17][15]
- Fastly’s reported Claude growth is not identified as ClaudeBot-only traffic.[10]
- Akamai’s 2026 commerce findings cover **July through December 2025**, while the 2026 Thales/Imperva report analyses **full-year 2025**.[18]

## How are pay-per-crawl and blocking changing?

Cloudflare introduced Pay Per Crawl in **private beta on 1 July 2025**. The system allows site owners to select crawlers to charge, set prices, manage payments through Stripe and monitor successful content deliveries.[19]

The documentation updated on **28 July 2026** describes a payment negotiation using HTTP headers. A crawler can present payment intent, or receive an **HTTP 402 Payment Required** response containing pricing information; successful payment can permit an **HTTP 200** response.[15]

The system charges again when the same page is crawled again, while errors are not billed. Paths such as `/robots.txt`, `/sitemap.xml`, `/security.txt`, `/.well-known/security.txt` and `/crawlers.json` remain free.[20]

Cloudflare expanded the configuration on **16 June 2026** with URI-pattern exclusions and dynamic pricing. Publishers can leave selected pages free while charging for others, and can set prices through an origin response header or a Cloudflare Worker.[21]

This is a documented technical mechanism, not evidence that the AI-crawling market has broadly adopted paid access. The cited product material does not quantify transactions, payments or publisher income.

Blocking controls have also changed over time. Cloudflare’s **1 July 2025** announcement introduced permission-based onboarding for new domains and said more than one million customers had already selected an earlier one-click blocking option.[22]

A later Cloudflare announcement dated **1 July 2026** described Search, Agent and Training controls for all customers and stated that new domains were scheduled to block Training and Agent traffic on ad-displaying pages from 15 September while allowing Search.[23]

However, the Cloudflare documentation updated on **2 November 2026** lists **Allow** as the new-domain default for Search, Agents and Training.[16][13] The cited documentation does not explain the difference between the announcement and the later setting.

Current controls distinguish **Block**, **Block on pages with ads**, **Allow** and **Disallow**. Cloudflare describes Disallow as a Training-only instruction conveyed through `robots.txt`, which remains voluntary unless additional edge enforcement is enabled.[16]

With Bot Preference Sync enabled, Cloudflare can enforce Disallow at the edge for training crawlers, but mixed-purpose Search-and-Training crawlers are excluded from that edge block.[16] Cloudflare separately reported in September 2026 that fewer than **1% of its sites blocked search crawlers**, compared with **17% blocking training**.[8]

## What do other 2026 measurements add?

Fastly’s January-May 2026 analysis found that AI requests were composed of approximately **85% crawlers and 15% fetchers** in May.[10][11] Fastly defines fetchers as traffic associated with user-directed requests, while crawlers represent more scheduled or automated activity.[10][11]

Fastly also reported Claude-related traffic at more than **555% above its January baseline** during the analysis period.[11] The source calls this “Claude traffic” and does not establish that the figure represents ClaudeBot alone.[4][3]

A separate Fastly investigation examined **23 September 2026** logs and found approximately **6.3 million requests from nearly 10,200 IP addresses** falsely claiming AI-crawler or fetcher identities.[24] The finding shows why a user-agent string alone cannot establish that a request came from the claimed operator.[1][24]

Akamai’s publishing and commerce measurements cover a different population. Its publishing report uses application-layer traffic observed through Akamai bot-management tools during **1 July-31 December 2025**.[17][18]

Akamai reported that commerce represented approximately **48% of AI-bot activity** and media represented **13%** in the relevant dataset.[17] Its July 2026 commerce announcement gave a more precise commerce figure of **47.9%** and said training crawlers accounted for more than **70% of AI-bot triggers in commerce** during that period.[18]

Imperva’s 2026 Thales Bad Bot Report analyses **full-year 2025** activity and reported that bots made up more than **53% of web traffic** while AI-driven bot attacks increased **12.5 times**.[25][9] The attack statistic should not be presented as legitimate GPTBot or ClaudeBot growth.[6]

## When should publishers use these measurements?

Publishers should match the metric to the decision:

- Use Cloudflare’s ratio when comparing observed HTML crawling with observed platform referrals, while stating the seven-day window and referral-header limitation.[4]
- Use Fastly’s figures when discussing AI-request growth, origin exposure, crawler-versus-fetcher composition or impersonated bot traffic.[24][6]
- Use Akamai’s figures for sector-specific AI-bot exposure, not for all-web traffic or crawl-to-referral rankings.[18]
- Use Imperva’s figures for automation and bad-bot context, not as a measure of useful crawler demand.[6]
- Use Cloudflare’s policy documentation when describing current blocking controls and Pay Per Crawl’s documentation when describing the closed-beta payment mechanism.[9]
- Use the academic study when discussing user click behaviour and possible search substitution, not when calculating crawler efficiency.[2]

The central distinction is between **requests, crawls, fetches, referrals, clicks, visits and revenue**. The cited evidence measures different points in that chain and does not justify collapsing them into one “AI traffic” number.[12][7]

## What is the background to GPTBot and ClaudeBot traffic?

An AI crawler may retrieve a page for several purposes, including search discovery, answer generation, user-directed fetching or model training.[8] Operators can also use multiple user agents, which means a platform-level measurement may include more than one bot identity.[4]

Cloudflare’s current ratio groups OpenAI-associated bots and Anthropic-associated bots at the platform level.[7][23] That is why a statement such as “GPTBot crawls 2,074 pages for every visitor” would overstate what the data proves: the published ratio applies to the OpenAI group and to observed referrals under Cloudflare’s methodology.[4]

Referral headers also do not capture every user action. Cloudflare notes that Claude’s native application does not include the relevant `Referer` header, and similar omissions may occur elsewhere.[4] The resulting ratio can therefore overstate the difference between crawling and referrals by an unknown amount.[4]


Background: what does “crawl-to-referral ratio” mean?

A crawl-to-referral ratio compares qualifying HTML crawl requests with HTML requests carrying a platform-associated referral hostname.[4][3] For example, a ratio of 2,073.96:1 means Cloudflare observed approximately 2,073.96 qualifying OpenAI-associated crawl requests for each qualifying OpenAI-associated referral in the stated measurement window.[4][3]

It does not mean that one platform fetched exactly that many unique pages, that every request reached a publisher’s origin, or that the platform generated no visits outside the observed referral-header method.[24][4] It also does not measure revenue or the value of a visit.[2]



## Timeline

| Date | Event | What it establishes |
|---|---|---|
| 1 July 2025 | Cloudflare announces Pay Per Crawl in private beta.[19] | A payment mechanism for selected crawler access was introduced; it was not evidence of market-wide adoption.[12][2] |
| July 2025 | Cloudflare publishes historical crawl-to-click data.[1] | Anthropic, OpenAI and Perplexity showed materially different monthly ratios during 2025.[1] |
| 1 July 2025 | Cloudflare announces permission-based AI-crawler onboarding for new domains.[22][2] | The announcement concerned new-domain onboarding and did not establish automatic blocking for every existing site.[2] |
| 16 June 2026 | Cloudflare adds advanced Pay Per Crawl configuration.[21] | URI exclusions and dynamic pricing become documented capabilities.[21] |
| 9 June 2026 | Fastly reports January-May AI-request growth.[11][10] | AI requests grew approximately 30%, around 6.5 times faster than human traffic.[11] |
| 28 July 2026 | Cloudflare Pay Per Crawl documentation is updated.[15] | The service is still described as closed beta.[15] |
| 23 September 2026 | Fastly measures false AI-bot identities in WAF logs.[24] | User-agent claims alone cannot authenticate crawler origin.[24] |
| 30 September 2026 | Cloudflare reports agent-request and crawler-purpose data.[12] | Daily AI-agent requests had grown more than 1,700% year over year; 52% of June 2026 crawler requests were classified as training.[8] |
| 2 November 2026 | Cloudflare access-policy documentation is updated.[5][13] | New-domain defaults are listed as Allow for Search, Agents and Training.[13] |
| 17 November 2026 | Cloudflare Radar snapshot is retrieved.[7] | The last-seven-day platform ratios are 3,351.73:1 for Perplexity, 2,073.96:1 for OpenAI and 456.23:1 for Anthropic.[1] |

## FAQ
### What is the AI crawler traffic ratio in 2026?
In Cloudflare Radar’s 17 November 2026 last-seven-day snapshot, Perplexity recorded **3,351.73 crawls per observed referral**, OpenAI recorded **2,073.96**, and Anthropic recorded **456.23**.[1] These are platform-level ratios based on Cloudflare’s HTML-request and referral-header methodology, not universal measures of visits or revenue.[1]

### What is the current GPTBot crawl ratio?
The cited November 2026 Cloudflare data does not publish a GPTBot-only ratio.[1][8] It reports an OpenAI platform-level ratio of **2,073.96 crawls per observed referral**, aggregating OpenAI-associated bots rather than isolating GPTBot.[3][1]

### Does ClaudeBot send less traffic than GPTBot?
The available data does not support that direct conclusion. Cloudflare’s chart displays ClaudeBot at **12.1%** and GPTBot at **8.1%** of an unspecified chart denominator, while its platform-level crawl-to-referral ratio is lower for Anthropic than OpenAI.[4][5] Those measures use different groupings and denominators.[12]

### Is Cloudflare Pay Per Crawl generally available?
No. Cloudflare documentation updated on **28 July 2026** still describes Pay Per Crawl as closed beta.[15] The cited documentation does not provide a general-availability date or aggregate data on paid requests and publisher payouts.[12][2]

### Does Cloudflare block AI crawlers by default in 2026?
The answer depends on the date and control being described. Cloudflare’s documentation updated **2 November 2026** lists Allow as the new-domain default for Search, Agents and Training, while earlier 2025 and 2026 announcements described permission-based or planned blocking policies.[13][16] The dated documentation should be cited instead of using an undated claim that Cloudflare blocks all AI crawlers by default.

## Sources
[1] The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — https://blog.cloudflare.com/crawlers-click-ai-bots-training
[2] Answering Without Referring: How AI Search Rewrites the Web’s Economic Bargain Qiaoni Shi, Kai Zhu, and Kai Gu Bocconi University, Milan, Italy — https://arxiv.org/html/2607.07652v1
[3] The 2025 Cloudflare Radar Year in Review- the rise of AI, post-quantum, and record-breaking DDoS attacks — https://blog.cloudflare.com/radar-2025-year-in-review
[4] The crawl before the fall… of referrals: understanding AI’s impact on content providers — https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar
[5] AI Insights | Cloudflare Radar — https://radar.cloudflare.com/ai-insights
[6] BAD BOT REPORT — https://www.imperva.com/resources/reports/2026-Thales-Bad-Bot-Report.pdf
[7] AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence — https://arxiv.org/html/2608.18352
[8] The Internet has a second audience — https://blog.cloudflare.com/agentic-web
[9] AI Bot Traffic: Which Bots to Allow or Block | Imperva — https://www.imperva.com/blog/ai-bot-traffic-which-bots-to-trust/
[10] AI Traffic Growth: Bots Outpace Humans  | Fastly — https://www.fastly.com/blog/ai-traffic-grew-6-5x-faster-than-human-traffic-this-year
[11] AI Traffic Grew 6.5x Faster Than Human Traffic This Year, Creating New Business Challenges and Opportunities - Tue, 06/09/2026 - 08:00 — https://investors.fastly.com/news-releases/news-release-details/ai-traffic-grew-65x-faster-human-traffic-year-creating-new
[12] The Impact of LLMs on Online News Consumption and Production — https://arxiv.org/html/2512.24968v2
[13] From Googlebot to GPTBot: Who’s crawling your site in 2025 — https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025
[14] Akamai Research: AI Bots Threaten the Foundation of Web-Based Business Models | Akamai — https://www.akamai.com/newsroom/press-release/akamai-research-ai-bots-threaten-foundation-of-web-based-business-models
[15] What is Pay Per Crawl? · Cloudflare AI Crawl Control docs — https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/?ref=technikatsu.com
[16] AI Crawler and Agent access policies — https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots
[17] AI Bot Defense and Monetization for Publishers | Akamai — https://www.akamai.com/site/en/documents/state-of-the-internet/2026/publishing-ai-botnet-report.pdf
[18] Akamai Research: Commerce Becomes the Epicenter for AI Bot Attacks and Agentic Fraud in 2026 — https://www.akamai.com/newsroom/press-release/akamai-research-commerce-becomes-the-epicenter-for-ai-bot-attacks-and-agentic-fraud-in-2026
[19] Introducing Pay Per Crawl (private beta) — https://developers.cloudflare.com/changelog/post/2025-07-01-pay-per-crawl
[20] Pay Per Crawl FAQ — https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq
[21] Pay Per Crawl advanced configuration — https://developers.cloudflare.com/changelog/post/2026-06-16-pay-per-crawl-advanced-configuration
[22] Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large;  Permission-Based Approach Makes Way for A New Business Model | Cloudflare — https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large
[23] New options to manage AI traffic — https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options?trk=public_post_comment-text
[24] AI Bot Cosplay | Fastly — https://www.fastly.com/blog/ai-bot-cosplay
[25] AI-driven Bot Attacks Surged 12.5x According to Thales Bad Bot Report - Company — https://www.imperva.com/company/press_releases/ai-driven-bot-attacks-surged-12-5x-according-to-thales-bad-bot-report/
[26] Your site, your rules: new AI traffic options for all customers — https://blog.cloudflare.com/content-independence-day-ai-options
