What is Crawl budget?
Nati Elimelech·Updated 27.07.2026 Crawl budget is the set of URLs Google can and wants to crawl on a site, set by crawl capacity and crawl demand.
Google’s definition splits crawl budget into two components. “Can” is crawl capacity, the ceiling on how many requests the site absorbs. “Wants” is crawl demand, how much the content interests Google in the first place. On small sites the budget is almost never a constraint; on large ones it decides which pages get crawled today and which wait. The term and this split are Google’s frame; other engines manage crawling with their own tools, like Bing’s Crawl Control.
How crawl budget works
Crawl capacity is set on both ends: fast, stable responses raise the ceiling, 5xx and 429 errors lower it, and Google’s own resources are finite too, divided across every site on the web. Crawl demand is set by the content: the size of the URL inventory, how popular the pages are, and how stale the content has gone. Actual crawling is what Google both can and wants to do: a fast server does not move a site Google finds uninteresting, and in-demand content is not crawled past what the server absorbs.
Two things that are easy to miss: Google defines a site for budget purposes as a unique hostname, so www.example.com and code.example.com have separate budgets. And the ceiling is shared across all of Google’s crawlers, so high demand from one of them leaves less capacity for the others.
Google aims its guidance on the subject at large sites: a million or more unique pages updating about weekly, or ten thousand or more updating daily, and by Google’s own wording these are rough estimates, not exact thresholds. Most sites never get near those sizes.
An example of crawl budget waste
A search engine spends most of its crawling on a site refetching known URLs and discovering new ones. In a commerce catalog this breaks down: every combination of filter and sort produces another URL, so ?color=red&sort=price and ?sort=price&color=red are two URLs for the same page. Googlebot burns most of its visits on the variations while new products wait in line. That is exactly what Google’s documentation lists as waste: duplicate content, the same page in different sort orders, soft 404s and redirect chains.
How crawl budget relates to SEO and GEO
A page that is not crawled does not enter any search engine’s index, and what is not in the index does not exist in results. On a large site, crawl budget sets the pace: the difference between an article indexed in minutes and one waiting days is usually budget burned on worthless URLs. The same URL inventory faces the crawlers building AI engines’ search indexes, like OpenAI’s OAI-SearchBot: a page they never reached does not enter the index those answers rest on.
What crawl budget is not
Crawl budget is not a quality score and not a ranking factor. It is how Google divides its crawling time between sites, derived from crawl capacity and crawl demand, and it cannot be raised directly. You can waste a huge budget on an excellent site and use a small one efficiently. What decides the outcome is what Google spends its visits on: a small budget spent well beats a large one burned on parameter URLs.
Questions about Crawl budget
Is crawl budget a ranking factor?▼
Crawl budget is not a ranking factor. It decides which pages Google reaches and when, not where they rank. But a page that never gets crawled never enters the index, and what is not indexed cannot rank at all.
When should you actually worry about crawl budget?▼
By Google's own guidance, crawl budget becomes a consideration on sites with a million or more unique pages whose content changes about once a week, or on sites with ten thousand or more pages whose content changes daily. Google itself qualifies these as rough estimates, not exact thresholds.
An early signal that applies at any size is a large share of URLs sitting in Discovered - currently not indexed in Search Console: Google knows those URLs exist and is choosing to postpone crawling them.
How do you spot crawl budget waste?▼
Crawl budget waste shows up in server logs and the Crawl Stats report: parameter pages, redirect chains, soft 404s and duplicate URLs getting crawled again and again. On a site already at its crawl capacity limit, every one of those requests comes at the expense of a page that matters.
How do you increase crawl budget?▼
Crawl budget grows through the two levers Google documents, and there is no way to directly ask Google for more crawling. Crawl capacity rises through the server: fast, stable responses raise the ceiling, and 5xx and 429 errors lower it.
Crawl demand rises through the content: more popular pages, content that genuinely gets updated, and a URL inventory free of duplicates.
Does a slow server shrink crawl budget?▼
A slow server shrinks crawl budget, and this is documented: Google writes that when a site slows down or responds with server errors, the capacity limit goes down and Google crawls less. The reverse is documented just as clearly, and the ceiling adjusts automatically over time, which makes response time the most direct lever on the server side.
Does noindex save crawl budget?▼
noindex does not save crawl budget: Google has to fetch the page to see the tag, and only then drops it, which Google itself describes as wasted crawling time. Blocking in robots.txt does prevent the fetch itself.
Google adds an important qualifier: budget freed up by blocking does not automatically move to other pages; it only helps if the site is already at its crawl capacity limit. That makes robots.txt a tool for cleaning up URL inventory, not for speeding up the crawl of content that matters.