Pop quiz hotshot: Is robots.txt a formally accepted internet standard? a) No b) Yes Most educated SEO’s will say “no,…
Differentiated Delivery: Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper
Differentiated Delivery : Stopping the bot storms: Serve trusted crawlers like Google a permissive robots.txt while blocking rogue scrapers with dynamic robots.txt in Apache, Nginx…
Cloudflare Enters the Robots.txt Fray with a “Content Signals Policy” for AI bots
Cloudflare just introduced a new way to tell bots what they can do with your content after they fetch it.…
Perplexity Says an Agent isn’t a Bot and Not Required to Honor Robots.txt
After Cloudflare posted yesterday about Perplexity using crawlers with stock agent names and not checking robots.txt, Perplexity has responded publicly.…
Tracking OpenAI – ChatGPT Bots – A Fresh Guide for Webmasters, Site Owners, and SEO’s
OpenAI runs multiple crawlers and user agents, and they do different jobs. Treating them as one single “AI bot” causes bad decisions. Below is the practical breakdown for webmasters, site owners, and SEO teams,
Updated 2026 – Googles Full List of Googlebot IP Addresses Spider IP Addresses
Google provides a list of four JSON files for the the IP address it’s crawlers use. This post explores that and how to track GoogleBot.
Google Talks About AI Search Controls, but it is Really Talking-Point Positioning
Google published a fresh post about “controls” for AI-powered Search features, but nothing new was announced. No tools shipped. No…
The Google Takes Aim at SerpAPI
On December 19, 2025, Google announced a lawsuit accusing SerpApi of “unlawful scraping” of search results and alleged violations of…
Cloudflare for SEO: A Deep Dive into Bot Management, AI Crawler Control, and Pay-Per-Crawl
For years, the web operated on a crawl, index, return visit cycle: search engines would crawl pages, send users back…
Beyond Marketing: The Cross-Functional Blueprint That Turns Search into Strategy
The Purpose of this Article Installment I have seen SEO grow into a powerhouse, having first become intrigued by making…
Cloudflare’s Content Signals: The Right Problem, An Unlikely Solution
The open web has long depended on an implicit exchange: publishers make content freely available, and platforms like Google send…
Negative and Link Based SEO Seems to be Still Working in Site Links
Cyrus Shepard led a weekend thread that has SEOs re-examining how much Google really discounts bogus backlinks. The discussion started…
Bots vs. Your Website: Which Ones to Allow and Which to Block?
Artificial intelligence is advancing rapidly, forcing website owners to face a big question: which AI crawlers should you exclude from…
Microsoft Shares Exact Signals AI Uses to Choose Content for Search Answers Optimization
Microsoft posted new recommendations to help website owners and marketers make their content more eligible for inclusion in AI-generated answers…
Introducing the RSL Standard: A New Licensing Layer for an AI-First Web
For years, SEOs and site owners have had to set by and watch their content get scraped, trained on, and…
Hands-On ChatGPT SEO Techniques from Top Practitioners
Recently at Pubcon Austin a group of SEO’s sat around at lunch and compared AI stories. Things like how we…
CloudFlare Says Perplexity is Crawling with Stock Agent – Its Monday
Cloudflare’s recent post calls out Perplexity for operating undeclared web crawlers, quietly sidestepping robots.txt while using a stock chrome agentname.…
Seznam.cz: Search Engine Overview
While Seznam.cz is the leading native search engine in the Czech Republic, it commands around 11–15% of the local search…
Googles 5 Options and How it Decided to Force Websites into AI Usage
Before Google trained on half the web anyway, it floated six opt-out tiers that would have let publishers throttle indexing,…
Half the Web Opted Out of Google AI Training! Google Trained Anyway.
A court filing just confirmed what many web publishers feared: Google’s opt-out tool for AI training doesn’t apply to Search AI. Despite 50% of content being excluded from Gemini training, Google still used that same data to train its search-based models—unless publishers blocked indexing entirely.
Google Doesn’t have to care
In freshly disclosed testimony during Google’s antitrust trial, a Google DeepMind exec confessed that Google continues training some of its…
Server Side Site Auditing and Optimization With AI – Masters of Traffic Part 4
Running a technical SEO audit can at times feel a bit overwhelming. AI simplifies the workflow process by giving you…
SearchGPT and ChatGPT Referral Tracking : Why your Analytics Can Be Missing It
With the news yesterday that ChatGPT is getting 400million visits a week, Webmasters are itchy and scratchy for a taste…
12 Essential Files SEOs Should Run Through ChatGPT for an SEO Tune-Up
LLM’s like Claude and ChatGPT are very good at SEO related information – it is like the new SEO super…
SEO Legends : “RepoMonkey Bait & Tackle/v1.01”
This is the place to dust off some of the “back-n-the-day” stories that are often told around the fireplace at…
OpenAI – ChatGPT – SearchGPT Web Crawlers Spiders and Bots
Last updated: 3/1/2026 OpenAI uses web crawlers and user agents to perform actions for its products, either automatically or triggered…