List of Google's common crawlers
Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically. The general technical properties of Google's crawlers also apply to the common crawlers.
The common crawlers generally crawl from the IP ranges published in the
googlebot.json object, and the reverse DNS mask
of their hostname matches crawl-***-***-***-***.googlebot.com
or
geo-crawl-***-***-***-***.geo.googlebot.com
.
The following list shows the common crawlers, their user agent strings as they appear in the HTTP
requests, their user agent tokens for the User-agent:
line in robots.txt, and the
products that are affected by crawl preferences for the crawler. Some crawlers have more than one
user agent token; you need to match only one crawler token for a rule to apply. The list is not
exhaustive, it only covers the requestors that are more likely to show up in log files and that
we've received questions about.
-
Googlebot
-
User-Agent
in HTTP requestsGooglebot Smartphone Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Rarely:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt User-agent token in robots.txt Googlebot
Example robots.txt group user-agent: Googlebot allow: /archive/1Q84 disallow: /archive
Affected products Crawling preferences addressed to the Googlebot
user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover. Googlebot Image
-
User-Agent in HTTP requests Googlebot-Image/1.0
robots.txt User-agent token in robots.txt Googlebot-Image
Googlebot
Example robots.txt group user-agent: Googlebot-Image allow: /archive/1Q84 disallow: /archive/moons.jpg
Affected products Crawling preferences addressed to the Googlebot-Image
user agent affect Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented. Googlebot Video
-
User-Agent in HTTP requests Googlebot-Video/1.0
robots.txt User-agent token in robots.txt Googlebot-Video
Googlebot
Example robots.txt group user-agent: Googlebot-Video allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the Googlebot-Video
user agent affect video-related Google Search features and other products dependent on videos. Googlebot News
-
User-Agent in HTTP requests Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings. robots.txt User-agent token in robots.txt Googlebot-News
Googlebot
Example robots.txt group user-agent: Googlebot-News allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the Googlebot-News
user agent affect all surfaces of Google News (for example, the News tab in Google Search and the Google News app). Google StoreBot
-
User-Agent in HTTP requests Desktop agent Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
Mobile agent Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
robots.txt User-agent token in robots.txt Storebot-Google
Example robots.txt group user-agent: Storebot-Google allow: /archive/1Q84 disallow: /archive/konbini
Affected products Crawling preferences addressed to the Storebot-Google
user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping). Google-InspectionTool
-
User-Agent in HTTP requests Desktop agent Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)
Mobile agent Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)
robots.txt User-agent token in robots.txt Google-InspectionTool
Googlebot
Example robots.txt group user-agent: Google-InspectionTool allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the Google-InspectionTool
user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products. GoogleOther
-
User-Agent in HTTP requests Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
robots.txt User-agent token in robots.txt GoogleOther
Example robots.txt group user-agent: GoogleOther allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the GoogleOther
user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. It has no effect on Google Search or other products. GoogleOther-Image
-
User-Agent in HTTP requests GoogleOther-Image/1.0
robots.txt User-agent token in robots.txt GoogleOther-Image
GoogleOther
Example robots.txt group user-agent: GoogleOther-Image allow: /archive/1Q84 disallow: /archive/moon.jpg
Affected products Crawling preferences addressed to the GoogleOther-Image
user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs. GoogleOther-Video
-
User-Agent in HTTP requests GoogleOther-Video/1.0
robots.txt User-agent token in robots.txt GoogleOther-Video
GoogleOther
Example robots.txt group user-agent: GoogleOther-Video allow: /archive/1Q84 disallow: /archive
Affected products Crawling preferences addressed to the GoogleOther-Video
user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs. Google-CloudVertexBot
-
User-Agent substring in HTTP requests Google-CloudVertexBot
robots.txt User-agent token in robots.txt Google-CloudVertexBot
Googlebot
Example robots.txt group user-agent: Google-CloudVertexBot allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the Google-CloudVertexBot
user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products. Google-Extended
-
User-Agent in HTTP requests Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. robots.txt User-agent token in robots.txt Google-Extended
Example robots.txt group user-agent: Google-Extended allow: /archive/1Q84 disallow: /archive/
Affected products Google-Extended
is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models that power those products. Grounding with Google Search on Vertex AI does not use web pages for grounding that have disallowed Google-Extended. Google-Extended does not impact a site's inclusion or ranking in Google Search.
A note about Chrome/W.X.Y.Z in user agents
Wherever you see the string Chrome/W.X.Y.Z in the user agent
strings in the table, W.X.Y.Z is actually a placeholder that represents the version
of the Chrome browser used by that user agent: for example, 41.0.2272.96
. This version
number will increase over time to
match the latest Chromium release version used by Googlebot.
If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.