Open the logs of a large site and you will find a lot more than one Googlebot. There is GoogleOther, there is Google-InspectionTool, and more recently Google-Agent and Google-GeminiNotebook.
Some of those names feed Search, some feed Gemini, and some feed nothing at all and just serve testing tools.
I went through the logs of several large sites to see which of these names actually show up. Some arrive daily. Some I have never seen. And two of the tokens in the documentation have no user agent at all, so they will never appear in a log.
OpenAI and Anthropic run their own crawlers on a similar pattern. Google’s list is longer, and the split between its bots is less intuitive.
Google’s three crawler categories
Each category behaves differently towards robots.txt, and that is the practical difference between them.
Common crawlers
The regular crawlers. The documentation says they always obey robots.txt when crawling automatically. This is where Googlebot and all its variants sit, along with GoogleOther and Google-CloudVertexBot.
Special-case crawlers
Crawlers tied to specific products, mainly advertising. They may ignore robots.txt, and they come from separate IP ranges.
User-triggered fetchers
Requests a user initiated. The documentation says these generally ignore robots.txt, precisely because the user asked for them.
Google’s bots that feed Search, including AI Overviews
A token is the name you write in the User-agent line of robots.txt to address a specific bot. It is not the same as the full user agent string that shows up in your logs, which is much longer, and some tokens have no string of their own at all.
| Token | What it feeds |
|---|---|
Googlebot | Search, including Discover and every Search feature |
Googlebot-Image | Images, Discover, and anywhere images, logos and favicons appear |
Googlebot-Video | Video features in Search |
Googlebot-News | Google News and the News app |
Storebot-Google | Google Shopping across all its surfaces |
Googlebot’s user agent, mobile and desktop:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36Googlebot-Image and Googlebot-Video arrive with short strings of their own, Googlebot-Image/1.0 and Googlebot-Video/1.0.
Googlebot-News is a control token only. It has no user agent of its own, and crawling happens with the regular Googlebot strings.
The same category also holds Google-InspectionTool, which does not crawl content for Search but serves the Search Console testing tools. The documentation says it has no effect on Search whatsoever.
AI Overviews and AI Mode are fed by Googlebot. They have no separate token. Google writes that AI is built into Search and integral to how Search functions, and that robots.txt directives for Googlebot are therefore the control over how a site is crawled for Search.
Google’s AI-related tokens: Google-Extended is not a crawler
Google-Extended
Google-Extended is not a crawler.
The documentation states plainly that it has no user agent of its own, and that crawling is done with Google’s regular user agents while the token serves in a control capacity. On the documentation page, where every other bot has a string, its entry simply says there is none.
It will never generate a request in your logs, and it will not change your crawl load. It governs what Google is permitted to do with content it has already fetched.
Per the definition, it covers two things: training of future generations of Gemini models that power Gemini Apps and the Vertex AI API, and grounding, meaning feeding content from the Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI.
The documentation goes on to say that Google-Extended does not affect a site’s inclusion in Search and is not used as a ranking signal.
What that means in practice: blocking Google-Extended does not remove you from AI Overviews or AI Mode. On a separate page Google writes that AI is built into Search, that robots.txt for Googlebot is the control over it, and points Google-Extended at its other systems.
GoogleOther
GoogleOther is a generic crawler. The documentation defines it as not tied to any specific product, and says various Google teams may use it to fetch publicly accessible content, for example one-off crawls for internal research and development.
Its user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36It has two variants, GoogleOther-Image and GoogleOther-Video, arriving as GoogleOther-Image/1.0 and GoogleOther-Video/1.0, both of which also honour the GoogleOther token.
It is commonly described as Google’s AI crawler. I searched its official definition for the words AI, Gemini and training, and none of them are there.
That does not mean the documentation rules it out. “Various product teams” and “internal research and development” is a definition that covers almost anything. What can be said with confidence: no Google document connects GoogleOther to Gemini training. If a documentation update names a product, that changes.
Google-CloudVertexBot
Google-CloudVertexBot is a real crawler, with a user agent, that shows up in logs. The documentation defines it as serving crawls that site owners themselves requested for building Vertex AI Agents, and says it has no effect on Search.
The documentation does not publish a full string for it, only the substring Google-CloudVertexBot inside the user agent, which is what to search for in logs.
It also falls back to the Googlebot token, so a site blocking Googlebot is already blocking it too.
User-triggered: Google-Agent and Google-GeminiNotebook
Google-Agent
Google-Agent is the agent that browses sites and performs actions on a user’s request. It runs on Google infrastructure, has its own IP list, user-triggered-agents.json, and its reverse DNS is google-proxy-***.google.com.
Its desktop user agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36Google is trialling the web-bot-auth protocol on it using the agent.bot.goog identity, signing a subset of requests cryptographically. The signature lets you confirm a request genuinely came from Google, and it is still experimental.
Google-GeminiNotebook
Google-GeminiNotebook fetches URLs that Gemini Notebook users added as sources for their projects. Unlike Google-Agent, it comes from the fetcher files rather than the agents file, and its reverse DNS ends in gae.googleusercontent.com.
Its desktop user agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)This name is new. Until July 2026 it was Google-NotebookLM, and the documentation marks the old name as supported only until August 2026. If the old string is hardcoded in a firewall rule or a log parser, it is worth replacing now. The failure is silent: the traffic keeps arriving, the rule just stops recognising it.
The rest
The same category holds FeedFetcher-Google for RSS and WebSub feeds, Google-Read-Aloud for reading pages aloud, Google-Site-Verification for Search Console verification, GoogleMessages for link previews in chat, Google-Pinpoint for personal document collections in Pinpoint, and Google-CWS for the Chrome Web Store.
Google’s ad crawlers, and Google-Safety
AdsBot-Google and AdsBot-Google-Mobile check the quality of pages that ads point to. Mediapartners-Google is the AdSense crawler. APIs-Google relates to push notification delivery.
All four share one behaviour: the documentation says they obey their own token, but that the global wildcard token is not taken into account. A User-agent: * with Disallow: / does not touch them. They need an explicit group in their own name.
Google-Safety, which handles abuse crawling such as malware discovery, ignores robots.txt entirely.
robots.txt: what each token blocks
You can control each token separately:
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /It is worth knowing what each block costs:
| Block this | Here is what happens |
|---|---|
Googlebot | Google stops crawling, and in practice you lose your Search visibility. Google no longer sees the content on your pages, and most of them drop out of the results over time. Technically this is not deindexing: a URL with strong external links can linger in results without a description, but that is the exception rather than the rule. AI Overviews and AI Mode go too, since they run on the same crawler |
Google-Extended | Google will not use the content to train future generations of Gemini, and will not ground answers on it in Gemini Apps or Vertex. What has already been trained is not undone. Per the documentation your Search ranking is unaffected |
GoogleOther | Google teams cannot pull the content for generic crawls not tied to any product |
Google-CloudVertexBot | Vertex AI customers cannot build an agent that draws on your content |
Google-InspectionTool | Your own Search Console testing tools stop returning results. No other effect |
As for the user-triggered category, meaning Google-Agent and Google-GeminiNotebook, the documentation says it generally ignores robots.txt and does not present these tokens as blocking controls.
One rule applies to every bot: once a specific group exists in the file, for example User-agent: Googlebot, the bot obeys only that group and ignores the rules under User-agent: *. A specific group and a wildcard group never merge. Any path you want blocked has to be repeated inside the specific group.
How to spot Google’s bots in your logs
It is worth searching for the token names themselves. Version numbers change, so searching by name alone works better:
Googlebot
GoogleOther
Google-InspectionTool
Google-CloudVertexBot
Google-Agent
Google-GeminiNotebookGoogle-InspectionTool comes from tests you run yourself in Search Console, so it appears in bursts rather than continuously.
How to verify a Google bot against its IP ranges
A user agent is a text string. Anyone can send a request with Googlebot written in it.
Google publishes five JSON files of IP ranges, and you can verify any request against them. I keep a full list that updates daily with every range and verification instructions for iptables and Nginx.
Reverse DNS for a common crawler ends in googlebot.com, and it is worth cross-checking that result against the CIDR ranges in the matching file.
What to take from this
- Google has three crawler categories, and they differ mainly in how they treat robots.txt.
- AI Overviews and AI Mode run on Googlebot. They have no token of their own.
- Google-Extended is a permission token rather than a crawler, and it covers Gemini and Vertex.
- GoogleOther is documented as a generic crawler. No document connects it to Gemini training, and none rules it out either.
Google-NotebookLMbecameGoogle-GeminiNotebook, and the old name is supported only until August 2026.- Ad crawlers ignore the wildcard and need a group in their own name.
- A user agent alone verifies nothing. Always check the IP.
