When outages happen, there is usually little explaining done or done late on a Friday with 'take out the trash'…
CloudFlare
Cloudflare for SEO: A Deep Dive into Bot Management, AI Crawler Control, and Pay-Per-Crawl
For years, the web operated on a crawl, index, return visit cycle: search engines would crawl pages, send users back…
Differentiated Delivery: Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper
Differentiated Delivery : Stopping the bot storms: Serve trusted crawlers like Google a permissive robots.txt while blocking rogue scrapers with dynamic robots.txt in Apache, Nginx...
Cloudflare’s Content Signals: The Right Problem, An Unlikely Solution
The open web has long depended on an implicit exchange: publishers make content freely available, and platforms like Google send…
Cloudflare Enters the Robots.txt Fray with a "Content Signals Policy" for AI bots
Cloudflare just introduced a new way to tell bots what they can do with your content after they fetch it.…
Bots vs. Your Website: Which Ones to Allow and Which to Block?
Artificial intelligence is advancing rapidly, forcing website owners to face a big question: which AI crawlers should you exclude from…
CloudFlare Releases AI Crawl Control out of Private Beta
Cloudflare has released to general availability of AI Crawl Control, a novel feature that provides website owners with enhanced authority…
No - OpenAI is Not Using Google for General Search Index
Lots of speculation (wishful thinking), alleging that OpenAI and ChatGPT is using Google for their search index. We've been fighting…
Perplexity Says an Agent isn't a Bot and Not Required to Honor Robots.txt
After Cloudflare posted yesterday about Perplexity using crawlers with stock agent names and not checking robots.txt, Perplexity has responded publicly.…
Cloudflare CEO Pulls FIRE Alarm: Google AI Is Breaking the Web's Economy! Search Is Broken!
Ten years ago, for every two pages Google scraped, you got one visitor, today, it’s six pages scraped for one visitor. And with AI? It’s 250 to 1… or 6,000 to 1!
Mostly Free Captcha Alternatives Worth Trying
In light of Google's announcement that all reCAPTCHA keys must be migrated to a Google Cloud (with fees of $8…