Bonus: a plugin for you to tracker Googlebot from verified agents and official ip list to spot when GoogleBot lands on your site.
Official GoogleBot and Google Crawler IP Lists
Google provides a list of four JSON files for the the IP address it's crawlers use:
- Googlebot-specific list The common crawlers used for Google's products (such as Googlebot). They always respect robots.txt rules for automatic crawls. GoogleBot documentation.
- Google Special Case Crawlers like AdsBot. Crawlers that perform specific functions for Google products (such as AdsBot) where there's an agreement between the crawled site and the product about the crawl process. These crawlers may or may not respect robots.txt rules of the search engine land. Special-case documentation by Google.
- User Triggered Fetchers - Covers Google users triggered crawlers - documentation. Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user.
- User Triggerd by Google - Google actually never details what this one is for, but does give us a list of ips.
- A full General Google IP list - covers all Google crawlers (beyond just Googlebot).
These JSON files include subnet masks and ranges. If JSON parsing is unfamiliar, site owners can use IP-range calculators (e.g. calculator.net) to turn subnet data into usable IP ranges.
Google Crawlers: Guide to User-Agent Strings by Purpose & Agent and IP List
Understanding Google's 20+ crawlers for better SEO and site optimization
Google Crawlers Organized by Purpose
This guide breaks down Google's official crawlers into their 4 main purposes, listing all agents with their User-Agent strings and linking to official Google documentation.
1: INDEXING (Core Search)
These crawlers discover and index content for Google Search results.
Googlebot Desktop
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Desktop web rendering & indexing | Verify Googlebot |
Googlebot Smartphone
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Mobile-first indexing | Verify Googlebot |
2: DISCOVERY (Link & Sitemap Crawling)
These agents find new URLs through links, sitemaps, and feeds.
Googlebot-Image
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Googlebot-Image/1.0 |
Image discovery & indexing | Image Crawling |
Storebot-Google
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (compatible; Storebot-Google/1.1; +http://www.google.com/bot.html) |
Google Shopping product discovery | Product Structured Data |
3: RENDERING (JavaScript & Visual)
These crawlers render JavaScript and check visual elements.
Googlebot (Chrome-Lighthouse)
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
JavaScript rendering pipeline | JavaScript SEO |
Mediapartners-Google
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mediapartners-Google |
AdSense content rendering | AdSense Crawler |
4: SPECIALIZED (Testing & Services)
Specialized crawlers for testing, AMP, and Google services.
AdsBot-Google
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (compatible; AdsBot-Google (+http://www.google.com/adsbot.html)) |
Google Ads landing page validation | AdsBot Verification |
AdsBot-Google-Mobile
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/adsbot.html) |
Mobile Ads landing page validation | Mobile AdsBot |
Google-Extended
| User-Agent String | Purpose | Official Docs |
|---|---|---|
Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html) |
Extended crawling for analysis | Google Crawlers Overview |
AMP Cache (.*googleusercontent.com)
| User-Agent String | Purpose | Official Docs |
|---|---|---|
google-amp-cache-request |
AMP page caching & serving | AMP Cache |
COMPLETE CRAWLER REFERENCE TABLE
| Crawler Name | User-Agent String | Purpose Category | IP Verification | Official Docs |
|---|---|---|---|---|
| Googlebot Desktop | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Indexing | googlebot.com + google.com | Verify |
| Googlebot Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) ... (Googlebot/2.1) |
Indexing | googlebot.com + google.com | Verify |
| Googlebot-Image | Googlebot-Image/1.0 |
Discovery | googlebot.com + google.com | Images |
| Storebot-Google | Mozilla/5.0 (compatible; Storebot-Google/1.1; +http://www.google.com/bot.html) |
Discovery | storebot.google.com | Shopping |
| Mediapartners-Google | Mediapartners-Google |
Rendering | mediapartners.google.com | AdSense |
| AdsBot-Google | Mozilla/5.0 (compatible; AdsBot-Google) |
Specialized | adsbot.google.com | Ads |
| AdsBot-Google-Mobile | Mozilla/5.0 (iPhone; ... AdsBot-Google-Mobile) |
Specialized | adsbot.google.com | Mobile Ads |
| Google-Extended | Mozilla/5.0 (compatible; Google-Extended/1.0) |
Specialized | google.com | Extended |
Related Resources
- Crawler and bot ID forum at webmasterworld.
- Bonus: a plugin for to tracker Googlebot from verified agents and official ip list.
As the CEO and founder of Pubcon and WebmasterWorld., Brett Tabke has been instrumental in shaping the landscape of online marketing and search engine optimization. His journey in the computer industry has spanned over three decades and has made him a pioneering force behind digital evolution. Full Bio
Visit Pubcon.com