Our investigation into AI crawler behavior all began with a surprising inquiry from one of our clients: if we block…
RFC 9309: Robots.txt Is Now an Official IETF Internet Standard (Robots Exclusion Protocol)
Pop quiz hotshot: Is robots.txt a formally accepted internet standard? a) No b) Yes Most educated SEO’s will say “no,…
Differentiated Delivery: Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper
Differentiated Delivery : Stopping the bot storms: Serve trusted crawlers like Google a permissive robots.txt while blocking rogue scrapers with dynamic robots.txt in Apache, Nginx…
Cloudflare Enters the Robots.txt Fray with a “Content Signals Policy” for AI bots
Cloudflare just introduced a new way to tell bots what they can do with your content after they fetch it.…
Perplexity Says an Agent isn’t a Bot and Not Required to Honor Robots.txt
After Cloudflare posted yesterday about Perplexity using crawlers with stock agent names and not checking robots.txt, Perplexity has responded publicly.…
YouBot Is Bigger Than You Think: Who Uses You.com Search?
YouBot may look like another AI crawler jamming up your logs. Dig into where You.com’s search infrastructure is actually available,…
Google Webmaster? Google Gives Site Owners an AI Button, But Not a New Deal
Google’s new Search generative AI control is useful, overdue, and beyond limited – to the point of uselessness? Site owners…
Hidden Prompt Injection vs. Twelve AI Assistants: One Obeyed, Three Warned the User
Our investigation into prompt injection all began with a surprising inquiry from one of our clients. They had found a…
EntityMap Wants To Give AI Search A Sitemap For Meaning
EntityMap: A Sitemap For What Your Site Knows What is it? EntityMap is a proposed open standard that lets publishers…
Tracking OpenAI – ChatGPT Bots – A Fresh Guide for Webmasters, Site Owners, and SEO’s
OpenAI runs multiple crawlers and user agents, and they do different jobs. Treating them as one single “AI bot” causes bad decisions. Below is the practical breakdown for webmasters, site owners, and SEO teams,
Updated 10-8-2026 – Googles Full List of Googlebot IP Addresses Spider IP Addresses
Google provides a list of four JSON files for the the IP address it’s crawlers use. This post explores that and how to track GoogleBot.
Google Talks About AI Search Controls, but it is Really Talking-Point Positioning
Google published a fresh post about “controls” for AI-powered Search features, but nothing new was announced. No tools shipped. No…
The Google Takes Aim at SerpAPI
On December 19, 2025, Google announced a lawsuit accusing SerpApi of “unlawful scraping” of search results and alleged violations of…
Cloudflare for SEO: A Deep Dive into Bot Management, AI Crawler Control, and Pay-Per-Crawl
For years, the web operated on a crawl, index, return visit cycle: search engines would crawl pages, send users back…
Beyond Marketing: The Cross-Functional Blueprint That Turns Search into Strategy
The Purpose of this Article Installment I have seen SEO grow into a powerhouse, having first become intrigued by making…
Cloudflare’s Content Signals: The Right Problem, An Unlikely Solution
The open web has long depended on an implicit exchange: publishers make content freely available, and platforms like Google send…
Negative and Link Based SEO Seems to be Still Working in Site Links
Cyrus Shepard led a weekend thread that has SEOs re-examining how much Google really discounts bogus backlinks. The discussion started…
Blocking Robots : Which Ones to Allow and Which to Block?
AI is obviously advancing rapidly, forcing website owners to face a big question: which AI crawlers should you exclude from…