The machines
Who took the work
Named crawlers belonging to AI companies, what each of them asked for, how much of it we sent, and the one we stopped serving. Every column below comes from a single table over a single window.
Across 91 days that is about 479 served reads a day, spread across the agents below.
Room 04 of 10
Read this first
This room ranks our instrument , not machine appetite
Until 7 August 2026 our door-knock capture was gated to AI surfaces only, so ordinary crawler traffic against ordinary pages was never recorded. The raw access log sees far more than the instrument did. Both are published below so the shortfall can be measured rather than described.
| Agent | Recorded by this instrument | Recorded in the raw access log | Coverage |
|---|---|---|---|
| amazonbot | 21,796 | 27,856 | 78.2% |
| meta-externalagent | 10,105 | 93,113 | 10.9% |
| gptbot | 8,869 | 91,267 | 9.7% |
| claudebot | 8,439 | 10,164 | 83.0% |
| oai-searchbot | 7,348 | 8,848 | 83.0% |
| chatgpt-user | 6,638 | 8,921 | 74.4% |
| googleother | 6,346 | 4,338 | 146.3% |
| perplexitybot | 4,824 | 2,270 | 212.5% |
| bytespider | 496 | 3,016 | 16.4% |
| google-extended | 5 | 22 | 22.7% |
Sources: the citation table (Live) and a frozen analysis of the raw access log taken on 7 August 2026. Across that log there were 3,432,235 requests in total, of which 250,590 were named AI crawlers, 146,034 Googlebot and 21,801 bingbot.
The visitors' book
Every named agent, and what each was sent
One row per agent. Attempts, what was served, what carried no content, how many separate days it came back, how many distinct anchors it reached, and the volume of data it took. Every column is drawn from the same table over the same window, so attempts and the split across it reconcile by construction.
| Agent | Attempts | Served | No content | Days seen | Anchors reached | Data sent (MB) |
|---|---|---|---|---|---|---|
| amazonbot withdrawn from credit | 21,796 | 18,803 | 2,993 | 45 | 4,303 | 7271.5 |
| bingbot | 14,659 | 12,228 | 2,431 | 51 | 3,058 | 5915.3 |
| meta-externalagent | 10,105 | 5,622 | 4,483 | 47 | 4,538 | 2851.7 |
| gptbot | 8,869 | 3,030 | 5,839 | 53 | 5,060 | 761.2 |
| claudebot | 8,439 | 4,130 | 4,309 | 52 | 3,254 | 7972.3 |
| oai-searchbot | 7,348 | 6,961 | 387 | 52 | 2,719 | 2231.1 |
| chatgpt-user | 6,638 | 5,902 | 736 | 57 | 1,303 | 2349.3 |
| googleother | 6,346 | 2,394 | 3,952 | 48 | 2,501 | 911.9 |
| perplexitybot | 4,824 | 2,532 | 2,292 | 53 | 2,083 | 936.7 |
| bytespider | 496 | 439 | 57 | 33 | 51 | 75.6 |
| duckassistbot | 360 | 329 | 31 | 47 | 250 | 129.0 |
| ccbot | 33 | 26 | 7 | 5 | 32 | 7.1 |
| google-extended | 5 | 1 | 4 | 3 | 5 | 2.9 |
| perplexity-user | 5 | 1 | 4 | 4 | 5 | 1.4 |
| anthropic-ai | 3 | 1 | 2 | 2 | 3 | 2.2 |
| claude-web | 3 | 0 | 3 | 2 | 3 | 1.0 |
| cohere | 3 | 0 | 3 | 2 | 3 | 1.0 |
Source: the citation table. Window: 2 Jun 2026 to 9 Aug 2026. Rule: served means status 200 or 206 with bytes above zero. Freshness: Live. Served plus no content equals attempts on every row.
Traffic diet
How much each agent actually swallowed
The same agents ordered by the volume of data taken rather than by request count. An agent that asks rarely but takes whole pages is doing something different from one that probes constantly and takes little.
| Agent | Data sent (MB) | Served reads | Average per read (KB) | Days seen | Last seen |
|---|---|---|---|---|---|
| claudebot | 7972.3 | 4,130 | 1,976.7 | 52 | 9 Aug 2026 |
| amazonbot | 7271.5 | 18,803 | 396.0 | 45 | 3 Aug 2026 |
| bingbot | 5915.3 | 12,228 | 495.4 | 51 | 9 Aug 2026 |
| meta-externalagent | 2851.7 | 5,622 | 519.4 | 47 | 8 Aug 2026 |
| chatgpt-user | 2349.3 | 5,902 | 407.6 | 57 | 9 Aug 2026 |
| oai-searchbot | 2231.1 | 6,961 | 328.2 | 52 | 9 Aug 2026 |
| perplexitybot | 936.7 | 2,532 | 378.8 | 53 | 9 Aug 2026 |
| googleother | 911.9 | 2,394 | 390.1 | 48 | 9 Aug 2026 |
| gptbot | 761.2 | 3,030 | 257.3 | 53 | 8 Aug 2026 |
| duckassistbot | 129.0 | 329 | 401.5 | 47 | 9 Aug 2026 |
| bytespider | 75.6 | 439 | 176.3 | 33 | 9 Aug 2026 |
| ccbot | 7.1 | 26 | 279.6 | 5 | 1 Aug 2026 |
| google-extended | 2.9 | 1 | 2,969.6 | 3 | 1 Aug 2026 |
| anthropic-ai | 2.2 | 1 | 2,252.8 | 2 | 1 Aug 2026 |
| perplexity-user | 1.4 | 1 | 1,433.6 | 4 | 1 Aug 2026 |
| claude-web | 1.0 | 0 | 1,024.0 | 2 | 1 Aug 2026 |
| cohere | 1.0 | 0 | 1,024.0 | 2 | 1 Aug 2026 |
Source: the citation table, recorded byte counts. Window: 2 Jun 2026 to 9 Aug 2026. Freshness: Live. The average column is derived and is not a measured figure.
What was consumed
Which parts of the site they wanted
Requests grouped by the kind of page they landed on, and then the individual pages taken most often.
| Kind of page | Attempts | Served | Distinct agents | Share of attempts |
|---|---|---|---|---|
| wiki | 25,493 | 17,126 | 12 | 28.3% |
| product | 25,308 | 22,736 | 13 | 28.1% |
| recipe | 19,931 | 11,687 | 11 | 22.2% |
| rest-api | 7,709 | 4,986 | 10 | 8.6% |
| category | 3,880 | 1,112 | 10 | 4.3% |
| brand | 1,965 | 1,624 | 12 | 2.2% |
| machine-sitemap | 1,702 | 251 | 8 | 1.9% |
| ai-citations | 1,182 | 925 | 6 | 1.3% |
| ai-anchors | 808 | 729 | 9 | 0.9% |
| multi-format | 748 | 356 | 16 | 0.8% |
| rest-teas | 569 | 381 | 13 | 0.6% |
| buy | 291 | 272 | 7 | 0.3% |
| ai-structured | 117 | 92 | 9 | 0.1% |
| ai-discovery | 108 | 43 | 13 | 0.1% |
| llm-files | 72 | 43 | 11 | 0.1% |
| landing | 41 | 30 | 9 | 0.0% |
| ai-media | 8 | 6 | 5 | 0.0% |
Source: the citation table, route bucket. Window: 2 Jun 2026 to 9 Aug 2026. Freshness: Live.
| Page | Served reads | Distinct agents |
|---|---|---|
| /citation-anchors.xml | 828 | 4 |
| /anchors.xml | 715 | 4 |
| /wp-json/teas/v1/asklee/thread/ | 324 | 5 |
| /teas-landing-sitemap.xml | 121 | 4 |
| /recipes/ | 117 | 8 |
| /teas-repaired-wiki-sitemap.xml | 105 | 3 |
| /brand/tetley/ | 99 | 6 |
| /product/yorkshire-tea-gold-blend-80-tea-bags-250g/ | 98 | 8 |
| /wiki/pukka-deep-dive/ | 87 | 5 |
| /brand/pg-tips/ | 87 | 8 |
| /wiki/who-owns-twinings/ | 84 | 5 |
| /wiki/tea-person-thomas-twining/ | 73 | 5 |
| /brand/lipton/ | 71 | 5 |
| /brand/loyd/ | 69 | 6 |
| /product/tropical-sun-soursop-20-tea-bags-30g/ | 68 | 5 |
Source: the citation table, served reads only. Window: 2 Jun 2026 to 9 Aug 2026. Freshness: Live. Top fifteen rows; the full list is in the JSON mirror.
Blocked
The one we stopped serving
Amazonbot ignored the limits we published. Its reads were first withdrawn from credit, and it is now refused at the server. Its whole history stays on this page.
Corrected 8 August 2026: most of the reads recorded against Amazonbot after 30 June 2026 did not happen. Until today the tracker recorded a status code predicted from the URL path rather than the one the server returned. Amazonbot was already being refused, so those requests were failures; they were written down as successful reads. 16,475 rows are affected, against 16,719 attempts recorded from that date. Because the served-read rule selects on status code, every one of them passed as a citation, and on that strength a refused crawler ranked first by read volume on this instrument.
The tracker now records the real status. The affected rows have not been rewritten, because editing a log to make an earlier claim true would be a worse fault than the one it fixes. They stay in the table, they stay on this page, and they are excluded from every total by the withdrawn-from-credit rule. The "times we served it anyway" figure above therefore reads high, and the honest statement is that 3,973 of those reads happened before the block took effect and can be relied on.
Corrected 7 August 2026: this page previously described Amazonbot as blocked while our own server was still returning content to it. The withdrawal from credit was real; our block was not yet enforced. The refusal is now dual-mechanism and has been effective since 30 June 2026: a robots.txt Disallow that Amazonbot ignored, and a server-level refusal added on 3 August 2026 that it cannot ignore. Its whole history stays on this page rather than being quietly omitted.
Still outstanding. The refusal is enforced at our server, not at the edge in front of it. Some Amazonbot requests are still answered from the edge cache and never reach us, so we cannot claim the block is total and we do not. Nothing served from the edge appears in the figures above, which means those figures are a floor for what Amazonbot actually received.
