Differentiated Delivery: Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper

brett Brett Tabkeon
Differentiated delivery cloaking robots txt

Over at WebmasterWorld, our 2-3m+ page archive of user-generated content has always been a magnet for crawlers. In 2023 the probes turned into massive bot storms reaching 10+ million page views a day. We have always done the usual dance - blacklisting user-agents, rate-limiting, forced log-ins, even JavaScript gates - but the traffic kept morphing and increasing. In the middle of all that I was updating robots.txt, and I dug up an ancient Perl script I'd used back-in-the-day - dusted it off and looked at it with fresh eyes.

Webmasterworld logoYou see, about an eon ago, I put together a viral little experiment called the Robots.txt Blog. It was a blog published inside the webmasterworld.com robots.txt file, with each "post"  written as comments. The idea came out of a WebmasterWorld thread about serving different versions of robots.txt.

The whole thing was a tongue-in-cheek proof-of-concept to show how a robots.txt file could be cloaked to hide sensitive details from competitors. The SEO crowd loved it as no one had seriously considered cloaking their robots.txt before.

It felt like a stunt back then, but now the legion of rogue and abusive bots has made it more standard now. Seriously, cloaking your robots.txt might sound like a one-off trick or a weird edge case, but there are many big sites doing it now (looking at you Reddit). The rise of rogue, abusive, and aggressive (some AI driven) bots has made static robots.txt files nearly useless because you can't guess some random new bot name.

Google's extension of robots.txt to include the Allow directive, does help somewhat and many of these crawlers ignore the protocol entirely. However, rel="ugc" seems more like an invitation to hungry UGC AI bots to come abuse us. eg: if your public "test directory" is in a robots.txt disallow directive, all you have done is give the url to a competitor.

The reality is that many of the bots that are going to be bad are the new AI segment of bots (NPR, MIT, CloudFlare). Using the differentiated delivery method to give them a full robots.txt block, effectively allows you to block them before they crawl. After all, the current landscape is that you want to allow Google and Bing, and block the other 50 gazillion bots out there.

This method allows you to do that by serving Google a clean robots.txt, while serving a full ban to everyone else with the understanding that the really bad-boy-bots are not going to read your robots.txt. Also know that many of the AI bots don't even support the Google robots.txt extension Allow directive.

So what we are saying, is that cloaking - more accurately called differentiated delivery - isn't just about hiding disallowed paths anymore. It's about controlling crawl load, protecting proprietary content, and preserving bandwidth to a degree. It is an imperfect solution, but it is better than nothing at this point.

Why Cloak Your Robots.txt?

  • Defend against rogue crawlers and scrapers: Serving a decoy or restricted version of robots may help protect server bandwidth and sensitive directories. Most of the bigger ones will obey robots.txt, but some are crashing websites.
  • Hide competitive intelligence: Cloaking prevents competitors from seeing which directories, scripts, or experimental sections are being blocked or tested, keeping internal development and SEO strategy private.
  • Control crawl load by user-agent: Differentiated delivery lets you send more lenient crawl rules to trusted bots like Googlebot or Bingbot while throttling or blocking lesser-known or aggressive crawlers.
  • Prevent cache leakage and spoofing: Dynamic robots.txt scripts (via Perl, PHP, Python, etc.) can verify legitimate search engine IP ranges and disable caching, stopping malicious/abusive bots from spoofing user agents getting access restricted paths.

  • Evolve robots.txt into an active defense layer: Instead of a static suggestion file, a cloaked or scripted robots.txt becomes a programmable gatekeeper that can adapt rules in real time based on threat level, IP verification, or bot behavior.

So, if you've ever wanted to hide your robots.txt from competitors or deal with a negative link attack, this post walks through how to serve one version of robots.txt to the bots you trust and something entirely different to everyone else.

Methods to Cloak your Robots.txt

There are two ways to go about hiding your robots.txt from bad bots:

  1. You can serve specific agents a specific version of your robots.txt (has issues)
  2. Create a robots.txt that fires a script and processes your own robots.txt (preferred)

Htaccess1: Cloak Robots With .htaccess

To cloak your robots.txt with a simple .htaccess hack and serve two or more versions of your robots.txt, you can use:

# Turn on rewrite
RewriteEngine On

# --- Serve custom robots for Googlebot
RewriteCond %{REQUEST_URI} ^/robots\.txt$ [NC]
RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
RewriteCond %{DOCUMENT_ROOT}/google-robots.txt -f
RewriteRule ^robots\.txt$ /google-robots.txt [L,END]

# --- Serve custom robots for Bingbot
RewriteCond %{REQUEST_URI} ^/robots\.txt$ [NC]
RewriteCond %{HTTP_USER_AGENT} bingbot [NC]
RewriteCond %{DOCUMENT_ROOT}/bing-robots.txt -f
RewriteRule ^robots\.txt$ /bing-robots.txt [L,END]

# --- Fallback for everyone else
RewriteCond %{REQUEST_URI} ^/robots\.txt$ [NC]
RewriteCond %{DOCUMENT_ROOT}/robots-default.txt -f
RewriteRule ^robots\.txt$ /robots-default.txt [L,END]

Using this version, you create separate robots.txt for each bot and a default one. In the example "bing-robots.txt" or "google-robots.txt" are used respectively, and "robots-default.txt" to be your full robots.txt that everyone can see. I would choose filenames that cannot be guessed easily. Also mark the files (or better, put them in a directory) themselves as blocked by your final robots.txt.

The core limitation of this method is someone can simply agent spoof a bot to bypass the cloaking. The only way to counter this is by verifying the IP address (Reverse DNS Lookup), which is complex and requires the active code method. To counter that, use a direct read of Google IPs and GoogleBot hosts file from Google. See our previous story on how to read Googles IP's and Hostname files.

Dealing With Robots Caching Robots.txt

Then there is caching. You have to make sure that any caching is thwarted (especially if you are using CloudFlare):

# Make sure caches treat UA variants separately and serve as plain text
<IfModule mod_headers.c>
 <FilesMatch "^(robots\.txt|google-robots\.txt|bing-robots\.txt|robots-default\.txt)$">
  Header set Content-Type "text/plain; charset=utf-8"
  Header set Cache-Control "no-cache, no-store, must-revalidate"
  Header set Pragma "no-cache"
  Header set Expires "0"
  Header add Vary "User-Agent"
  Header add Vary "Accept-Encoding"
  Header unset Content-Location
 </FilesMatch>
</IfModule>

Nginx Alternative

If you are using Nginx instead of Apache:


map $http_user_agent $robots_file {
    default "robots-default.txt";
    "~*Googlebot" "google-robots.txt";
    "~*bingbot" "bing-robots.txt";
}

server {
    # ... other server config

    location = /robots.txt {
        # Assumes the robots files are in the server's root directory - Use 'alias' for files outside the server root, or 'rewrite'
        rewrite ^/robots.txt /$robots_file break;
    }
}

Scripting cloak icon2: Cloaking Robots.txt with Scripting

We have a couple different options when using scripting to generate a robots.txt. The overhead performance hit on this is minimal because a script to do this is very lightweight.

The first method to talk about is with active code. This entails turning your robots.txt file into a pure script that executes when it is requested. You can accomplish this a few different ways:

Option A : Active Code

This example executes a Perl script when a request for robots.txt is processed. It could easily fire a Python, PHP, or any code your server is enabled to execute as well.


# .htaccess in your web root
Options +ExecCGI
AddHandler cgi-script .cgi .pl RewriteEngine On RewriteRule ^robots\.txt$ /cgi-bin/robots.pl [L,QSA]

Option B - ScriptAlias in Apache Vhosts File

This one, maps one path to your script without rewrite rules. Use in vhost, or in .htaccess on hosts that allow it.


ScriptAlias /robots.txt /home/USER/cgi-bin/robots.pl
<Files "/home/USER/cgi-bin/robots.pl">
Options +ExecCGI
Require all granted

Option C - .txt File as CGI

This was the path I took on WebmasterWorld. This method is useful if you want the URL to literally hit a file named robots.txt, but it's a script.

# Put a file named robots.txt in your web root, but it's actually Perl
Options +ExecCGI
AddHandler cgi-script .txt

<Files "robots.txt">
SetHandler cgi-script
</Files>

SecuritySecurity Concerns

Configuring your server to execute .txt files as CGI scripts (Option C) introduces a minor but non-trivial security risk to be managed. This approach effectively tells your server to treat any .txt file as executable code, which could be an issue if an attacker manages to upload a malicious text file elsewhere on your server through a vulnerable upload form, compromised CMS, or insecure file permissions (which is more likely to happen with txt files). To mitigate the risk, you  should limit the executable scope using the <Files> directive to only the specific robots.txt file, ensure your CGI script has minimal file system permissions, validate. Additionally, place your actual robots script outside the web root when possible and use ScriptAlias (Option B) instead, as this provides better isolation.  Reminder, never enable ExecCGI globally or on entire directories containing user-uploadable content.

Robots.txt as Executable Code

Finally, here is an example code to fire your custom script when robots.txt is requested. (convert this to your favorite scripting lang with ChatGPT)

View Perl Code


#!/usr/bin/perl
use strict;use warnings;

# Read request info
my $ua = $ENV{HTTP_USER_AGENT} // '';
my $host = $ENV{HTTP_HOST} // '';
my $ip = $ENV{REMOTE_ADDR} // '';

# Example: serve a stricter robots to unknown bots
my $is_google = ($ua =~ /Googlebot/i);
my $is_bing = ($ua =~ /bingbot/i);

# Cache & content headers If you vary by UA, add Vary to prevent bad caches
print "Content-Type: text/plain; charset=utf-8\r\n";
print "Cache-Control: no-cache, must-revalidate\r\n";
print "Vary: User-Agent\r\n\r\n";

if ($is_google or $is_bing) {
print <<"ROBOTS";
User-agent: *
Allow: /

Sitemap: https://$host/sitemap.xml
ROBOTS
} else {
# Slightly stricter default
print <<"ROBOTS";
User-agent: *
Disallow: /private/
Disallow: /tmp/

Sitemap: https://$host/sitemap.xml
ROBOTS
}

View PHP Code
< ?php

// Read request info
$ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
$host = $_SERVER['HTTP_HOST'] ?? '';
$ip = $_SERVER['REMOTE_ADDR'] ?? '';

// Example: serve a stricter robots to unknown bots
$is_google = (stripos($ua, 'Googlebot') !== false);
$is_bing = (stripos($ua, 'bingbot') !== false);

// Cache & content headers - If you vary by UA, add Vary to prevent bad caches
header("Content-Type: text/plain; charset=utf-8");
header("Cache-Control: no-cache, must-revalidate");
header("Vary: User-Agent");
echo "\r\n"; // Extra line break after headers

if ($is_google || $is_bing) {
echo "User-agent: *\n";
echo "Allow: /\n";
echo "\n";
echo "Sitemap: https://$host/sitemap.xml\n";
} else {
// Slightly stricter default
echo "User-agent: *\n";
echo "Disallow: /private/\n";
echo "Disallow: /tmp/\n";
echo "\n";
echo "Sitemap: https://$host/sitemap.xml\n";
}

?>

View Python Code
#!/usr/bin/env python3
import os
import sys

def main():
# Read request info from environment variables
ua = os.environ.get('HTTP_USER_AGENT', '')
host = os.environ.get('HTTP_HOST', '')
ip = os.environ.get('REMOTE_ADDR', '')

# Example: serve a stricter robots to unknown bots
is_google = 'Googlebot'.lower() in ua.lower()
is_bing = 'bingbot'.lower() in ua.lower()

# Cache & content headers - If you vary by UA, add Vary to prevent bad caches
print("Content-Type: text/plain; charset=utf-8")
print("Cache-Control: no-cache, must-revalidate")
print("Vary: User-Agent")
print() # Empty line to separate headers from content

if is_google or is_bing:
print("User-agent: *")
print("Allow: /")
print()
print(f"Sitemap: https://{host}/sitemap.xml")
else:
# Slightly stricter default
print("User-agent: *")
print("Disallow: /private/")
print("Disallow: /tmp/")
print()
print(f"Sitemap: https://{host}/sitemap.xml")

if __name__ == "__main__":
main()

Test from Command Line Shell


curl -I https://example.com/robots.txt
curl -A "Googlebot" https://example.com/robots.txt
curl -A "SomeRandomBot" https://example.com/robots.txt

 

ScalesBest and Ethical Practices 2025

I have read the TOS of all the major related entities (Google, Bing, OpenAI, Yandex, CloudFlare) and cannot find anywhere that any of them ban or disallow this process. All of the discussion they talk of cloaking or differentiated delivery in reference to user based content. Even Google only says they are worried about the maintenance:

Avoid serving different versions of your robots.txt file to different requestors (in other words, cloaking), as this creates a maintenance burden, may prevent you from debugging crawl issues, or have otherwise unintended consequences.

In fact, asking Google AI about this practice lead to its endorsement for cloaking your sitemap:

You should ensure your sitemap is accessible to search engines by submitting it to Google Search Console, and you can prevent regular users from seeing the XML file by using standard web server configuration or a firewall rule. This is not considered cloaking and is a recommended practice for managing your sitemap.

In deeper chats with both Bing Copilot and Google Gemini/AIMode, they were very happy to show us how to hide are robots.txt and sitemap files.

Google ai mode robots txt sitemap

Even so, we cannot fathom a scenario where any major search engine would have issue with you implementing this. There is no way to use this as a 'black hat' technique for rankings or to manipulate a search engine. The only way this can be used is to protect a website from nasty bots that can take down websites.

Google cloaking ai chat
"Is it ok to cloak or hide my robots.txt and sitemap from users?"

Conclusion: The Necessity of a Dynamic Defense

The "Robots.txt Blog" was a clever response that highlighted the flexibility of server configuration. Today, the underlying technique of dynamic robots.txt generation has evolved from my SEO novelty or competitive tactic into a necessary defensive measure against the exponential growth of unchecked AI and aggressive scrapers.

The rise of rogue, abusive, and aggressive AI bots has rendered static robots.txt files increasingly ineffective, as these bots frequently ignore standard protocol. Cloaking - or more accurately, differentiated delivery is no longer just about hiding a list of disallowed paths from a competitor; it is about controlling server load, protecting proprietary content, and preserving bandwidth.

While simple HTTP cloaking via server rules (like Apache's RewriteRule or Nginx's map directive) provides a fast, basic layer of defense, it remains vulnerable to User-Agent spoofing. The most robust solution is to use active code (Perl, PHP, Python) to process the request, which allows for advanced verification steps like Reverse DNS Lookup to confirm if a requesting bot truly belongs to the claimed search engine (e.g., verifying "Googlebot" is coming from a legitimate Google IP range).

In this new era, robots.txt is evolving from a mere suggestion for polite crawlers to a first line of active defense against malicious and overzealous automation. The choice between static cloaking and a scripted solution is a balance between performance and security that site owners must now weigh based on the value of their content and the intensity of the bot traffic they face.

Related: