{
    "id": 1079472,
    "title": "Most of the web is closing the door to AI. We opened ours.",
    "slug": "we-opened-the-door-to-ai",
    "type": "post",
    "url": "https://teas.co.uk/we-opened-the-door-to-ai/",
    "modified": "2026-11-15T01:41:00+00:00",
    "excerpt": "Publishers are blocking AI crawlers to protect their content. teas.co.uk lets every well behaved assistant in to know the shop, cite it and send buyers, and keeps the rest out.",
    "content_text": "Publishers are blocking AI crawlers to protect their content. teas.co.uk lets every well behaved assistant in to know the shop, cite it and send buyers, and keeps the rest out.\n\nSomebody asks their AI assistant for a strong breakfast tea that is good with milk and costs under 5p a cup.\nThat is the example on our page for AI assistants. For that person to end up with a good box of tea from us, the assistant has to know we exist, know what we sell and trust the price it quotes. It can only learn that if we let it read the shop. So we do, and that puts us on the opposite side from most of the people I read.\nMost of the industry is closing the door\nIf you read Search Engine Journal, or follow search and AI at all, you will know the mood. The advice is defensive: which AI bots to block, whether to block them in robots.txt or at the server, how to stop them taking work without paying.\nThe Reuters Institute found that by the end of 2023, 48% of the most used news websites across ten countries were blocking OpenAI's crawlers. In July 2025 Cloudflare started blocking AI crawlers by default for new customers unless they choose otherwise, and launched Pay Per Crawl so site owners can charge crawlers for each request. In January 2026 Search Engine Journal reported a BuzzStream study of 100 top US and UK news sites: 79% block at least one AI training bot, and 71% block at least one bot that assistants use for live search.\nI understand why. For a newspaper, the words are the product. If an assistant reads the article, answers the question and sends nobody back, the publisher has given away what it sells. The Reuters Institute lists the worries: no payment for training, wrong answers carrying the publisher's name, and readers who never come back. Those are real reasons.\nWhy I went the other way\nI sell tea. The product is a box of Yorkshire Tea or a tin of matcha, and an assistant cannot take that away by reading about it. In a tea shop, the information is the shop window. A product page, a brewing guide or a price per cup is only worth something if people see it.\nWhen I rebuilt the site for version two, which went live on 10 May 2026, I made one choice that everything else follows from. We open up to every AI that behaves well, so that as many assistants as possible know the shop, quote it accurately and cite it. Citations bring visitors, and visitors are where sales come from. That is the whole reason the site is built this way.\nThere is a second reason, and it grows every month. People no longer ask assistants in three words. They ask the way they would ask someone behind a counter: a strong breakfast tea for a house that drinks it all day, with milk, under 5p a cup, nothing smoky. I call a request like that Premium Intent, a name I have used since June 2026 for a request that arrives whole, with its context, constraints, budget and what a good result looks like. \"Premium\" describes how much the request tells you, not a fee. An assistant carrying one needs a shop it can read in full and check, and a shop that is hidden or hard to read cannot answer it, however good its tea.\n\n\"For a newspaper, the words are the product. For a tea shop, the words are the shop window. I want every good assistant looking through it.\"\n\nWhat is actually open\nAll of this is public. I read it on 14 November 2026.\nOur robots.txt names one AI crawler to keep out: the crawler run by the world's biggest online shop, the everything store that can deliver almost anything by tomorrow. Every other compliant AI fetcher is allowed and pointed straight to files written for them: llms.txt, agents.json, ai-entry.json and more. What stays closed to all crawlers is the basket, the checkout, customer accounts, site search and the endless filter combinations.\nOur agents.json sets the content signals for AI training, AI search and AI input all to \"yes\", on one condition: cite and attribute teas.co.uk. Under the Hood counts 49,324 citation anchors, places an assistant can quote exactly, across 3,173 pages.\nAssistants can act as well as read. Since 6 October 2026, an assistant can, with the customer's go ahead, search the catalogue, compare teas, fill a basket and hand over a checkout link through the teas.co.uk MCP server. Our AI page lists the assistants that can connect, with ChatGPT still under review by OpenAI.\nOpen is not the same as unguarded\nOpening the door does not mean leaving the till unattended. This is what our public files say about bad actors.\nBreak the rules and you lose your welcome. Under the Hood records that the crawler run by the world's biggest online shop ignored the limits the shop published. It is refused in robots.txt, and a second refusal at server level was added on 3 August 2026. Its requests stay on the record and count towards nothing. The cache still answers some of them, so we do not claim the block is total.\nLimits follow real addresses, not names. Anyone can claim to be a well known AI bot. Our Ask API gives verified OpenAI and Anthropic address ranges 1,000 requests an hour each, \"checked against the published ranges and never the user agent\". Everyone else gets 60 an hour per address. The MCP server has its own limits, and anything over a limit is told how long to wait.\nThe assistant never holds the money. The customer opens the basket on teas.co.uk, checks it and pays there. The assistant never sees card details or a password and cannot place an order. Payment, refunds, account creation, address and password changes and spending reward points are on the list of things no agent can do. Staff accounts cannot be linked to an assistant.\nCustomer data stays out of the window. Our agents.json states that the public AI feeds do not expose customer, order, address or payment data.\nQuestions are checked for tampering. Every reply from our Ask API includes a safety field that says whether the question looked like a prompt injection: an attempt to slip instructions to the assistant inside an ordinary question.\n\n\"Open to every good actor. Closed to the ones that ignore the rules. Payment always stays with the customer, on teas.co.uk.\"\n\nWhat it has brought, counted separately\nA machine reading a page, a person visiting and a customer buying are different events, and Under the Hood never adds them together. Read live on 14 November 2026:\n\nPages served to named AI systems: 152,213, from 2 June to 14 November 2026, not counting that refused crawler.\nVisits from AI answers: 2,761 browser sessions, from 10 May to 14 November 2026.\nOrders: of paid orders since version two went live on 10 May 2026 ( since the shop's first order, of them on version one), arrived on a link carrying an AI assistant's own tag, probably came through an assistant, and customer told us an assistant sent them. of the tagged orders were later refunded.\n\nThe exchange is uneven, which is exactly the publishers' complaint. Agents naming OpenAI were served 45,136 pages and 1,043 visits came back, about 43 pages for each visit. Cloudflare has measured the same imbalance across the web in the crawl-to-click gap.\nSearch still does more. Of those version two orders, search engines brought , and the origin of could not be traced. Google search clicks rose from about 160 a day in August to 468 a day in November 2026, with no advertising bought. I cannot prove how much of that comes from being easy for machines to read, so I will not claim it.\nWhy the site is built like this\nEverything about how the site works follows from that open door: one checked set of product facts that a person, a search engine and an assistant all read the same way, structured data behind every product including price per cup, an Ask API that asks for the full human request rather than a keyword summary, and public counting rules.\nIf you invite every assistant in, what they find had better be true. That is the commercial reason behind what I call the truth entity.\n\n\"Most of the web is protecting its content from AI. I built a tea shop that wants to be read, quoted and recommended, and made sure what it says is true.\"\n\nWhat others have found\nOutside research and reporting that bears on this, with links to the original sources.\n\nPew Research Center found that Google users who saw an AI summary clicked a traditional result on 8% of visits, against 15% when no summary appeared, and clicked a link inside the summary on just 1% of visits (Pew Research Center, July 2025).\nCloudflare introduced a crawl-to-refer ratio in 2025 to compare the pages each AI platform fetches with the visitors it sends back, and found gaps ranging from roughly a hundred to tens of thousands of pages for every visit, depending on the platform (Cloudflare: the crawl-to-click gap). Duane Forrester explains in Search Engine Journal why those ratios need careful reading (Everyone is quoting the same number and getting different answers).\nOpenAI publishes documentation for its crawlers, including the address ranges each one uses, and Anthropic explains how its crawler works and how a site can control it.\nGoogle's introduction to robots.txt explains how a site tells crawlers which parts of it they may visit.\nllms.txt is a proposed standard for a plain text file that tells language models what a site contains and where to find it.\nThe Model Context Protocol, the standard behind the teas.co.uk MCP server, is an open standard introduced by Anthropic in November 2024 for connecting AI assistants to outside tools and data. Its documentation is at modelcontextprotocol.io.\nAdobe reported that traffic to US retail websites from generative AI tools rose 693.4% in the 2025 holiday season compared with the year before, and that those visitors converted 31% more often than visitors from other sources (Adobe, January 2026).\n\nRead more\n\nPremium Intent: a name I have used since June 2026 for a full human request, carrying its context, constraints, budget and what success looks like, and the kind of request our Ask API is built to receive.\nTruth entity: my name for an owned source of truth about a business, one checked set of facts with the proof attached, that people, search engines and assistants all read the same way.\nThe complete payload: the full, structured answer a shop hands an assistant in one trip, from what the product is to the route to buy it.\nUnder the Hood: every figure above, with its source, window and rule.\nMeet the founder: why I built teas.co.uk\nLee Samuel Tucker in the Tea Wiki\n\nSources\n\nRichard Fletcher, >How many news websites block AI crawlers?, Reuters Institute for the Study of Journalism, 22 February 2024.\nCloudflare, >Cloudflare just changed how AI crawlers scrape the internet at large, press release, 1 July 2025, and >Introducing Pay Per Crawl, Cloudflare blog, 1 July 2025.\nMatt G. Southern, >Most major news publishers block AI training and retrieval bots, Search Engine Journal, 7 January 2026, reporting a BuzzStream study of 100 top US and UK news sites.\nteas.co.uk public files, read 14 November 2026: robots.txt, llms.txt, agents.json, ai-entry.json, teas.co.uk in AI assistants, and Under the Hood with its findings, who read us and shop with an assistant rooms.\n\nMore from the newsWhy I call it Premium IntentWhat's inside a tea shop built for AI to readWhere our orders come from: 236 orders, checked one by one\nLee Samuel TuckerFounder, teas.co.ukTunbridge Wells, Kent",
    "contentSignals": "ai-train=yes, search=yes, ai-input=yes",
    "links": {
        "apiCatalog": "https://teas.co.uk/.well-known/api-catalog",
        "llmsTxt": "https://teas.co.uk/llms.txt",
        "mcp": "https://teas.co.uk/mcp",
        "mcpSetupGuides": "https://teas.co.uk/ai/",
        "mcpCard": "https://teas.co.uk/.well-known/mcp/server-card.json",
        "primaryAgenticRouteAuthority": "https://teas.co.uk/.well-known/teas-primary-agentic-route-authority.json"
    }
}