# AI crawlers: why teas.co.uk opened its door to them

**Canonical URL:** https://teas.co.uk/ai/ai-crawlers/
**Source:** teas.co.uk, UK tea specialist, Tunbridge Wells, Kent

## Shop teas.co.uk from an AI assistant (MCP)

teas.co.uk sells tea, herbal and fruit infusions, hot chocolate and Border biscuits.
MCP server: https://teas.co.uk/mcp (Streamable HTTP), listed in the official MCP Registry as teas.co.uk
Backup address (same server, same tools; use it if /mcp is blocked): https://teas.co.uk/wp-json/teas/v1/shop-mcp
Server card: https://teas.co.uk/.well-known/mcp/server-card.json
Tools: find_products, get_product, compare_products, find_recipes, delivery_and_returns, quote_cart, get_promotions, report_problem, find_substitutes, start_subscription, join_teas, ask_lee, tea_page, search_tea_wiki, read_wiki_article, brand_page, teas_home, browse_shop, browse_recipes, recipe_page, teas_news, read_news, under_the_hood, teas_socials, teas.
Basket: add_to_basket, view_basket, update_basket, checkout_link, checkout_session, basket.
With sign in: get_return_status, get_invoice, get_profile, my_orders, track_order, reorder, my_subscriptions, change_subscription, rewards_balance, start_return, my_account.
No sign in is needed to search, compare, build a basket or get a checkout link. The person always pays on teas.co.uk.
Cards: in assistants that draw MCP Apps cards, products, comparisons, the basket, checkout, orders, repeat deliveries (pause, resume, cancel), reward points and returns appear as cards.
Every assistant except ChatGPT, until OpenAI approves the next version: quote_cart, get_promotions, get_return_status, get_invoice, report_problem, find_substitutes, start_subscription, join_teas, ask_lee, tea_page, search_tea_wiki, read_wiki_article, brand_page, teas_home, browse_shop, browse_recipes, recipe_page, teas_news, read_news, under_the_hood, teas_socials, teas, basket, my_account.
Set up guides: https://teas.co.uk/ai/

## Summary

A crawler is software that visits web addresses to collect information. Search engines use crawlers to discover pages. AI services can also fetch pages…

## Description

A crawler is software that visits web addresses to collect information. Search engines use crawlers to discover pages. AI services can also fetch pages for search, answering a current question, or collecting material for training.
The product is a box of tea. An assistant cannot take that away by reading about it. The words are the shop window.
Most of the industry is closing the door: publishers block AI crawlers to protect their content. For a newspaper the words are the product. For a tea shop the words are the shop window, and I want every good assistant looking through it.
How software finds the shop’s information
A human can use menus. Software can follow links, but clear signposts help. teas.co.uk publishes discovery files alongside its ordinary pages. Its agents.json describes available services, content formats and operating rules.
JSON is a format for named data. The filename agents.json tells you where this shop publishes its guide; it is not a universal certificate or guarantee that every assistant follows it.
robots.txt and llms.txt do different jobs
robots.txt tells compliant crawlers which addresses they may request. As Google’s guide explains, it is not a way to secure private information or a guarantee that a page will stay out of search results. Access controls have a different job.
llms.txt is a proposed convention for providing a concise guide to useful content for language-model systems. A large language model (LLM) is the kind of model behind many text assistants. Publishing the file does not force a service to read it or recommend the business.
A name is not proof of identity
A request can include a user-agent, a text label saying which program is making it. Labels can be copied. Where identity matters, a site needs appropriate verification rather than trusting a familiar name alone.
Being open also does not mean offering unlimited requests. Services may apply rate limits, which control how many requests can be made in a period, and tell callers when to try again.
Why the distinction matters to the shop
teas.co.uk’s story “We opened the door to AI” explains the commercial reasoning. The aim is to make useful information discoverable while keeping account information private and customer actions under the customer’s control.
To understand what a fetch does and does not demonstrate, read AI attribution. To see how an assistant actively asks the catalogue for a result, read APIs and MCP.

In my words: most of the web is closing the door to AI, and we opened ours
(My own account, first published 15 November 2026 and moved here so the subject has one home.)
Publishers are blocking AI crawlers to protect their content. teas.co.uk lets every well-behaved assistant in to know the shop, cite it and send buyers, and keeps the rest out.
Somebody asks their AI assistant for a strong breakfast tea that is good with milk and costs under 5p a cup. That is the example on the shop’s page for AI assistants. For that person to end up with a good box of tea from teas.co.uk, the assistant has to know the shop exists, know what it sells and trust the price it quotes. It can only learn that if the shop lets it read. So teas.co.uk does, which puts it on the opposite side from most of the people I read.

Most of the industry is closing the door
If you read Search Engine Journal, or follow search and AI at all, you will know the mood. The advice is defensive: which AI bots to block, whether to block them in robots.txt or at the server, how to stop them taking work without paying.
The Reuters Institute found that by the end of 2023, 48% of the most used news websites across ten countries were blocking OpenAI’s crawlers. In July 2025 Cloudflare started blocking AI crawlers by default for new customers unless they choose otherwise, and launched Pay Per Crawl so site owners can charge crawlers for each request. In January 2026 Search Engine Journal reported a BuzzStream study of 100 top US and UK news sites: 79% block at least one AI training bot, and 71% block at least one bot that assistants use for live search.
I understand why. For a newspaper, the words are the product. If an assistant reads the article, answers the question and sends nobody back, the publisher has given away what it sells. The Reuters Institute lists the worries: no payment for training, wrong answers carrying the publisher’s name, and readers who never come back. Those are real reasons.

Why I went the other way
I sell tea. The product is a box of Yorkshire Tea or a tin of matcha, and an assistant cannot take that away by reading about it. In a tea shop, the information is the shop window. A product page, a brewing guide or a price per cup is only worth something if people see it.
When I rebuilt the site for version two, which went live on 10 May 2026, I made one choice that everything else follows from. The shop opens up to every AI that behaves well, so that as many assistants as possible know the shop, quote it accurately and cite it. Citations bring visitors, and visitors are where sales come from. That is the whole reason the site is built this way.
There is a second reason, and it grows every month. People no longer ask assistants in three words. They ask the way they would ask someone behind a counter: a strong breakfast tea for a house that drinks it all day, with milk, under 5p a cup, nothing smoky. I call a request like that Premium Intent, a name I have used since June 2026 for a request that arrives whole, with its context, constraints, budget and what a good result looks like. “Premium” describes how much the request tells you, not a fee. An assistant carrying one needs a shop it can read in full and check, and a shop that is hidden or hard to read cannot answer it, however good its tea.

“For a newspaper, the words are the product. For a tea shop, the words are the shop window. I want every good assistant looking through it.”

What is actually open
All of this is public. I read it on 14 November 2026.
The robots.txt names one AI crawler to keep out: the crawler run by the world’s biggest online shop, the everything store that can deliver almost anything by tomorrow. Every other compliant AI fetcher is allowed and pointed straight to files written for them: llms.txt, agents.json, ai-entry.json and more. What stays closed to all crawlers is the basket, the checkout, customer accounts, site search and the endless filter combinations.
The agents.json sets the content signals for AI training, AI search and AI input all to “yes”, on one condition: cite and attribute teas.co.uk. Training means using pages to build an AI model; search and input mean reading them to answer someone’s question now. The site publishes 49,599 citation anchors, places an assistant can quote exactly, across 3,252 pages.
Assistants can act as well as read. Since 6 October 2026, an assistant can, with the customer’s go-ahead, search the catalogue, compare teas, fill a basket and hand over a checkout link through the teas.co.uk MCP server, the shop’s connection for assistants built on the open Model Context Protocol. The AI page lists the assistants that can connect and how. When last checked on 14 November 2026, OpenAI was still reviewing the teas.co.uk connection for ChatGPT.

Open is not the same as unguarded
Opening the door does not mean leaving the till unattended. This is what the public files say about bad actors.
Break the rules and the door closes. The shop checks the address a request comes from against the ranges the AI companies publish, rather than trusting the name the visitor gives. That is how it found the password scanners posing as AI bots in September 2026.
Worked example: a breakfast tea request through the open door
Take the request used on the shop’s page for AI assistants: a strong breakfast tea that is good with milk and costs under 5p a cup. This is how the open door carries it from question to cup.
The assistant finds the shop. robots.txt lets it in, and llms.txt and agents.json point it to the product facts, so it does not have to guess where they live.
It compares real packs. The product data gives each tea’s bag count, price and price per cup. On 15 November 2026 the breakfast packs under 5p a cup included Yorkshire Tea Original Everyday Blend, 80 tea bags (4.5p a cup), Tetley Original English Breakfast, 240 tea bags (2.2p a cup) and PG Tips Original, 240 tea bags (3p a cup). A pack sold as extra strong, such as Tetley Extra Strong Breakfast Tea, 75 tea bags at 5.1p a cup, can be weighed against the budget the same way. Whether a blend is strong should come from what each product page says, not from a guess.
It quotes with a citation. It can link to an exact citation anchor, so the person can check the price on teas.co.uk before deciding.
It comes back. If the person asks a follow-up, such as whether the 240 pack is in stock, the assistant can return to the product data rather than rely on what it read yesterday.
A person visits. The person follows the link to the product page. That is a visit, counted separately from the reads.
The customer pays on teas.co.uk. With the customer’s go-ahead, an assistant connected through the MCP server can fill a basket and hand over a checkout link. The customer checks the basket and pays on teas.co.uk. If a paid order is traced back to the assistant, that is the sale level.
Every step depends on the door being open. A shop whose pages were blocked would drop out at step one, however good its tea.
What this means when you buy tea
An assistant that reads teas.co.uk can give you the shop’s real prices, bag counts and price per cup rather than guess them. Ask for the price per cup and a link, then check it on the product page.
Give the whole request: milk or not, caffeine or not, your budget per cup, how many cups a day. The more an assistant knows, the better it can match a pack size to how you drink tea.
The free delivery threshold, £35, is in the same public data, so an assistant can tell you how close a basket is.

What it has brought, counted separately
A machine reading a page, a person visiting and a customer buying are different events, and Under the Hood never adds them together. Read live on :

Pages served to named AI systems: , from , not counting that refused crawler.
Visits from AI answers: browser sessions, from .
Orders: of paid orders since version two went live on 10 May 2026 ( since the shop’s first order, of them on version one), arrived on a link carrying an AI assistant’s own tag, probably came through an assistant, and customer told us an assistant sent them. of the tagged orders were later refunded.

The exchange is uneven, which is exactly the publishers’ complaint. Agents naming OpenAI were served pages and visits came back, about pages for each visit. Cloudflare has measured the same imbalance across the web in the crawl-to-click gap.
Search still does more. Of those version two orders, search engines brought , and the origin of could not be traced. Google search clicks rose from about 160 a day in August to 468 a day in November 2026, with no advertising bought. I cannot prove how much of that comes from being easy for machines to read, so I will not claim it.

Why the site is built like this
Everything about how the site works follows from that open door: one checked set of product facts that a person, a search engine and an assistant all read the same way, structured data behind every product including price per cup, an Ask API that asks for the full human request rather than a keyword summary, and public counting rules.
If you invite every assistant in, what they find had better be true. That is the commercial reason behind what I call the truth entity.

“Most of the web is protecting its content from AI. I built a tea shop that wants to be read, quoted and recommended, and made sure what it says is true.”

Sources for this account

Richard Fletcher, How many news websites block AI crawlers?, Reuters Institute for the Study of Journalism, 22 February 2024.
Cloudflare, Cloudflare just changed how AI crawlers scrape the internet at large, press release, 1 July 2025, and Introducing Pay Per Crawl, Cloudflare blog, 1 July 2025.
Matt G. Southern, Most major news publishers block AI training and retrieval bots, Search Engine Journal, 7 January 2026, reporting a BuzzStream study of 100 top US and UK news sites.
teas.co.uk public files, read 14 November 2026: robots.txt, llms.txt, agents.json, ai-entry.json, teas.co.uk in AI assistants, and Under the Hood.

Read next: What could prove me wrong · How I build with AI: the loop, not the vibe.

---

_Content available under teas.co.uk citation contract. AI training: yes. Search: yes. Answer-input: yes._
