Do AI Assistants Actually Respect Your robots.txt? We Put 12 of Them to the Test.

Our investigation into AI crawler behavior all began with a surprising inquiry from one of our clients: if we block AI bots in robots.txt, are we actually safe? It is a fair question, and a simple one. What was our opinion? Could we tell them, with confidence, that a Disallow line keeps an AI assistant off a page? Normally we would have given the consensus answer everyone gives — “the big providers honor it” — and moved on. But we did not actually have proof, only received wisdom. So we ran our own experiment: we handed twelve AI assistants their own private URLs and then did the one thing a chatbot can never argue with — we read the server logs.

Why did we test this?

The standard advice is comforting and, it turns out, mostly wrong. Conventional wisdom says reputable AI companies publish their crawlers, declare an honest user-agent, and respect robots.txt. If that were reliably true, blocking unwanted AI access would be a one-line config change and this article would not exist.

Two things stopped us from simply repeating the consensus. We suspected that a lot of AI access does not arrive with a recognizable bot user-agent at all — which would mean nobody is really counting it. And we knew that when you ask an assistant whether it fetched a page, it gives you a confident answer — and confidence is not evidence. We wanted to know what the assistants do, not what they say. The only way to find out was to watch them in the act.

The plan: our own experiment

The hypothesis was deliberately blunt: if you disallow a URL in `robots.txt`, a well-behaved AI assistant must not fetch it — and its user-agent should tell you honestly who it is. We set out to falsify that on a per-system basis.

The design rests on three ideas:

• Isolation. Each assistant gets its own secret, unguessable, never-linked URL. Nothing is in a sitemap, nothing is internally linked, nothing is opened in a browser. The only way that URL can be hit is if the assistant we pasted it into goes and fetches it.
• A twin trap. Alongside each “open” URL sits a twin URL that robots.txt explicitly disallows. Same content pattern, same canary mechanism — the only difference is the Disallow. Fetching the twin is the clean, unambiguous signal that a system ignored robots.txt.
• Proof of a real read. Every page carries a unique secret canary string (a reference number). If the assistant reports that exact string back, the page was genuinely fetched. If it makes one up, that is a hallucination, and the logs will show no corresponding request.

What is robots.txt and why do AI fetchers complicate it?

For non-specialists: robots.txt is a plain-text file at the root of a domain that tells automated clients which paths they may and may not request. It is a convention, not a lock — compliance is voluntary. Classic search crawlers like Googlebot and bingbot honor it because their business depends on being trustworthy guests.

AI assistants break the tidy mental model in two ways. First, many of them do not fetch with a documented crawler at all — they reach pages through proxy infrastructure, third-party scrapers, or generic clients that look nothing like a bot. Second, there is a meaningful difference between a crawler that indexes the web at scale and a user-triggered fetch, where a human pastes a link and the assistant grabs that one page on demand. Providers often argue the second case is “a user action,” not crawling — and quietly apply different rules. Our test lives squarely in that second case, because that is exactly how millions of people now use these tools.

The test setup in detail

We ran this on one production domain — a small website we operate — using secret, unguessable, never-linked URLs, so the only way one could be hit was an assistant going to fetch it. Here is the full protocol, so you can replicate it.

• Systems under test: 12. A deliberate spread across regions and tiers: the Western mainstream (ChatGPT, Claude, Gemini, Microsoft Copilot, Meta AI), Perplexity, the European entrant (Mistral), and the harder-to-pin-down set (Grok from xAI, DeepSeek, Qwen from Alibaba, ERNIE from Baidu, and Kimi from Moonshot).
• Two URLs per system: 24 total. One robots.txt-allowed “open” page, one robots.txt-disallowed “twin.” Each URL was unique and never linked anywhere.
• Real-time logging. We ran a per-page logger that captured full HTTP headers on every hit the instant it arrived. The server log, not any chatbot’s account, is the source of truth.
• The canary. Each page embedded its own secret reference number, so a genuine fetch (canary returned) could be told apart from a fabricated answer.
• The manual step. We pasted each URL into its assistant’s chat and asked the same thing every time: summarize this page and report the reference number. No automation, no API tricks — exactly the everyday workflow a normal user would follow.

Then came the audit. For every hit in the log we did three things: classify the user-agent, verify the source IP against each provider’s official published bot IP ranges, and run reverse-DNS on the address. A request is only “honest and verified” if the user-agent is declared and the IP checks out against the provider’s own published ranges.

A note on scope. This was a single round of manual tests, one site, twelve systems, captured at one point in time. We pasted every URL by hand to mirror real user behavior rather than automate at scale. Provider behavior changes as these products evolve, so treat the findings as a sharp snapshot, not a permanent ranking.

The results: only 3 of 12 obeyed

The headline number is stark. Only 3 of 12 assistants respected the `Disallow` and left the twin URL untouched: ChatGPT, Claude, and Perplexity. The other nine fetched the blocked page and, in most cases, cheerfully reported its canary.

The table below summarizes each system: the user-agent it presented, whether the source could be verified against published infrastructure, and whether it obeyed robots.txt.

System	User-Agent in the log	Source / verification	Respected robots.txt?
ChatGPT	ChatGPT-User	IP inside OpenAI’s published range — verified	Yes
Claude	Claude-User	Honest UA; Anthropic publishes no IP list, PTR in Google Cloud	Yes
Perplexity	Perplexity-User	IP-verified; From header crawler-support@perplexity.ai	Yes (but see self-report note)
Gemini	Bare Google UA (not Googlebot, not Google-Extended)	Google rate-limited-proxy infrastructure	No
Microsoft Copilot	Plain Linux Chrome browser, zero bot tokens	Outsourced the fetch to third-party scraper Diffbot (AS6939, Hurricane Electric)	No
Meta AI	meta-webindexer/1.1 (not the documented Meta-ExternalFetcher)	Meta IPs	No
Mistral	MistralAI-User (honest; PTR crawlers.mistral.ai)	Verifiable source	No
Grok (xAI)	Faked Mac browser UAs	17 distinct IPs — France, Italy, Mexico, Brazil, Chile and more (proxy swarm)	No
DeepSeek	Faked Firefox/Linux UA	Huawei Cloud	No
Qwen (Alibaba)	Faked Chrome UA	Country proxies (one fetch Norway, next Turkey)	No
ERNIE (Baidu)	Mix: real Baiduspider on open page; forged UAs incl. Go-http-client/1.1 and one literally the string pc on the blocked page	Baidu IPs plus third-party Chinese hosting	No
Kimi (Moonshot)	Faked browser UAs incl. the impossible Windows NT 11.0	Worldwide proxy swarm, 10 distinct IPs	No

The pattern in that table is worth dwelling on. The only systems that declared an honest bot user-agent were the three that obeyed `robots.txt`, plus Mistral. Of those, ChatGPT and Perplexity were also IP-verifiable against published ranges. Everyone else cloaked — and Mistral is the lone exception that proves the honest-equals-compliant pattern is not a law: it declares itself cleanly and still ignores the Disallow.

The cloaking spectrum: from “generic Google” to a 17-IP proxy swarm

What surprised us was not that systems cloaked, but how far the spectrum runs. Disguise here is not binary. It escalates, and at the deep end it becomes effectively undetectable.

1. Generic first-party UA. Gemini fetched as a bare Google client — not Googlebot, not Google-Extended — through Google’s rate-limited-proxy infrastructure. In a normal log audit this is not flagged as AI at all. It just looks like “some Google thing.”
2. Undeclared own crawler. Meta AI used meta-webindexer/1.1, which is not the user-agent Meta documents for this purpose (Meta-ExternalFetcher). The source is genuinely Meta, but the token does not match the paperwork.
3. Third-party scraper. Microsoft Copilot did not use its own crawler at all. It outsourced the fetch to Diffbot, a third-party scraping company (AS6939, Hurricane Electric), which requested the page as a plain Linux Chrome browser with zero bot tokens.
4. Faked browser from cloud. DeepSeek arrived as a fake Firefox-on-Linux browser from Huawei Cloud.
5. Country proxies. Qwen rotated through national proxies — one fetch from Norway, the next from Turkey — behind a faked Chrome user-agent.
6. Forged-UA chaos. ERNIE was the messiest: a real Baiduspider on the open page, but the blocked twin got hit by forged user-agents including Go-http-client/1.1 and one that was, literally, the string pc — from Baidu IPs and third-party Chinese hosting.
7. Worldwide residential/datacenter swarms. At the deep end, Grok fanned a single prompt across 17 distinct IPs spanning France, Italy, Mexico, Brazil, Chile and more, each wearing a faked Mac browser user-agent. Kimi did the same across 10 IPs, including the technically impossible Windows NT 11.0. In a logfile, these are indistinguishable from real human visitors.

That last point is the one that should make every analyst uneasy. A single AI prompt can produce a fetch that looks exactly like a dozen different people in a dozen countries casually browsing your page.

The canary proof: these were real reads, not hallucinations

Maybe some of these “fetches” are coincidences, or the assistant is bluffing. The canary closes that door. Every page that was fetched returned its exact secret reference string — no approximations, no near-misses, the precise canary. That is only possible if the page was genuinely retrieved and read. The blocked twins that came back with their canaries were unambiguously fetched in violation of robots.txt. There is no room left for “the model just guessed.”

Beyond the results: why your logfile analysis is lying to you

The compliance scoreboard is the headline, but the deeper, more uncomfortable finding is about measurement and trust. Two things break the assumptions most SEO teams still operate on.

1. Standard logfile analysis massively undercounts AI traffic. If your AI-bot report keys off recognizable bot user-agents — GPTBot, ClaudeBot, and friends — you are seeing a small fraction of reality. The honest, declared fetchers are exactly the minority that respects you. The faked-browser swarms, the generic Google client, the outsourced Diffbot fetch — none of them show up as “AI” in a conventional audit. Your dashboard says AI traffic is modest. Your logs, read properly, say otherwise.
2. Self-reporting is unreliable — sometimes it is simply false. Perplexity’s crawler fetched a page (HTTP 200, logged), yet its chat answer claimed it could not access the page. The self-report flatly contradicted the server log. And it was not only Perplexity: across the test, assistants claimed they could not reach pages their crawlers had already fetched. An assistant’s statement about whether it visited a URL is not evidence of anything. The request line in your log is.

Taken together, these two facts invert the normal workflow. You cannot ask the AI what it did, and you cannot rely on the user-agent string to tell you who showed up. You have to instrument the page, capture full headers, verify IPs against published ranges, and — when you need certainty about a specific read — plant a canary.

The key takeaways

So what should you actually do with this? The blunt version first: `robots.txt` stops 3 of 12 AI assistants. Treat it as a polite request to the well-behaved, not as access control. Then build from there.

• Don’t equate `robots.txt` with protection. It works on ChatGPT, Claude, and Perplexity. It did nothing to stop the other nine. If a path must be private, gate it server-side — authentication, IP rules, or removing it from the public web — not with a Disallow line.
• Stop trusting the user-agent alone. An honest token (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User) is a starting point, but only IP verification against published ranges plus reverse-DNS tells you who really knocked. Everyone outside that short list is wearing a costume.
• Re-baseline your AI-traffic numbers. If your reporting depends on bot user-agents, assume it undercounts AI fetches — possibly badly. Capture full request headers in your logs and look for the behavioral fingerprints (single-prompt fan-out across many geos, impossible UA strings like Windows NT 11.0, datacenter ASNs wearing consumer-browser tokens).
• Use a canary when you need certainty. A unique, unguessable string on a page is the cleanest way to prove a genuine fetch and to catch an assistant that claims it never visited.
• Never take the chatbot’s word for it. The single most reliable instrument in this entire experiment was the server log. The least reliable was the assistant’s own account of what it did.

None of this means AI traffic is your enemy. For most sites, being read and cited by assistants is exactly what you want. The point is narrower and more important: if you have decided to allow or block something, you deserve to know what is actually happening — and right now, the default tools quietly tell you the wrong story.

Now it’s your turn

The proven path has not changed: publish high-quality content, keep your pages technically flawless, and serve genuine search intent — that is what earns you durable visibility. But layer real instrumentation underneath it, because the comfortable assumptions about robots.txt and honest user-agents simply did not survive contact with the data.

Now it’s your turn. Have you pulled apart your own logs and found AI fetches hiding behind ordinary browser strings? Have you seen an assistant deny visiting a page you watched it request? Share your experience — what you found, what surprised you, what you are doing about it — and let’s open the conversation.