What is robots.txt?
Nati Elimelech·Updated 27.07.2026 robots.txt is a text file that says which URLs crawlers may fetch. robots.txt controls crawling, not indexing.
robots.txt is a single text file at the root of a site, and it is the first thing a bot asks for before it touches any page. Its rules apply only to the exact address the file sits on: that host, that protocol (http or https) and that port. So every subdomain needs its own file.
How robots.txt works
What robots.txt is built from
The file is built out of groups. Each group opens with a user-agent line naming the bot it is talking to, followed by disallow and allow lines with a path. A bot finds the group that belongs to it and follows those rules. The ordinary use is keeping bots off paths that have nothing for them, like internal search results and filter pages generated from parameters. On a large site that is exactly where crawl budget leaks away.
Who sets the robots.txt rules
The standard behind the file is RFC 9309, published in 2022, which formalized a protocol that had existed since 1994. RFC 9309 states outright that the rules in it are not a form of access authorization: this is a request a bot can honor or ignore, not a lock. Google publishes its own interpretation on top of the standard and caches the file for up to 24 hours, so an edit does not take effect immediately.
What robots.txt does not cover
The file generally does not govern user-triggered fetchers, which arrive because someone asked just now rather than as part of a crawl. And it simply does not apply to anyone who decides not to honor it, because it is not a law.
A robots.txt example
The rule that is easiest to miss is that only one group applies to any given bot. Google puts it as “Only one group is valid for a particular crawler”, and the bot picks the group whose user-agent most specifically matches its own:
User-agent: *
Disallow: /search
User-agent: Googlebot
Disallow: /tmpGooglebot reads the second group only. The /search block written in the asterisk group does not apply to it at all, and it will crawl /search freely. The two groups are never merged.
How robots.txt relates to SEO and GEO
robots.txt sits in front of everything else. A single disallow line stops the bot from fetching the page, and a page that is never fetched has no content to put in the index and nothing to rank on. Nothing further down the chain repairs a fetch that did not happen, not good content and not inbound links.
That same file is served to the other search engines’ bots and to AI engine bots. OpenAI documents that GPTBot honors it, so the same line that keeps a page’s content out of the index also keeps it out of the answers that page could have been cited in.
The difference between robots.txt and noindex
robots.txt stops crawling. noindex stops indexing. The difference turns practical the moment you try to hide a page from results, and it runs both ways. Google writes that a page blocked in the file can still be indexed and shown in search without a description, because the URL is discovered through external links. And in reverse: a bot blocked from crawling the page never sees the noindex tag written inside it, and Google explicitly requires that the page not be blocked in robots.txt for that rule to work. Getting a page out of results means allowing the crawl and adding noindex, or putting it behind a password.
Questions about robots.txt
What happens if my robots.txt returns a 5xx error?▼
A robots.txt returning 5xx makes Google stop crawling for the first 12 hours while it keeps retrying the fetch. After that it crawls from the last good cached copy for up to 30 days, and with no cached copy it assumes there are no crawl restrictions. After 30 days, if the site itself is reachable, Google treats it as having no robots.txt; if the site has availability problems, Google stops crawling.
RFC 9309 is stricter, and not entirely consistent either: section 2.3.1.4 says an unreachable file must be treated as a complete disallow, while section 2.4 permits using a cached copy beyond 24 hours precisely when the file is unreachable. The 12 hours and the 30 days are Google numbers, not the standard.
The clearest gap is the no-cached-copy case: Google assumes no restrictions where the standard requires a complete disallow.
What happens if my robots.txt returns a 404?▼
A robots.txt returning 404, like any 4xx except 429, is treated by Google as if no valid file existed, which means no crawl restrictions at all. RFC 9309 likewise lets a crawler access any resource in that state. Google handles 429 as a server error. A file dropped in a deploy or a path that changed erases every block in the file at that moment, with no warning.
I blocked a page in robots.txt and it still shows in Google. Why?▼
robots.txt blocks the crawl, not the URL. In Google's words: it will not crawl or index content blocked by robots.txt, but it might still find and index a disallowed URL if it is linked from other places on the web, and that result will not have a description.
Removing a page from the index means allowing the crawl and adding noindex.
Does robots.txt support wildcards?▼
robots.txt supports two wildcards: the asterisk, which stands for zero or more characters, and the dollar sign, which marks the end of the URL. RFC 9309 defines both as characters bots must support, and Google documents that it, Bing and other major search engines do support them. The line disallow: /*.pdf$ blocks crawling of every PDF on the site.
Is robots.txt case sensitive?▼
robots.txt is case sensitive in some parts of a line and not others, which is a common source of mistakes. Field names and the user-agent value are case-insensitive, so Googlebot and googlebot are the same.
The path in disallow and allow lines is case-sensitive, so disallow: /Admin does not block /admin. The filename itself must also be robots.txt in lowercase, per RFC 9309.
Do all bots respect robots.txt?▼
Not all bots respect robots.txt, for two separate reasons.
The first is that nothing enforces it: Google writes that the instructions in the file cannot enforce crawler behavior, so a malicious bot simply ignores them.
The second is a documented exemption: Google writes that user-triggered fetchers generally ignore robots.txt rules, and OpenAI writes of ChatGPT-User that because these actions are initiated by a user, robots.txt rules may not apply.