What is Crawl capacity?
Nati Elimelech·Updated 27.07.2026 Crawl capacity is Google's ceiling on the total time a server holds connections open for it.
Google throttles itself. It does not want to take your server down, so it puts a ceiling, which its documentation also calls hostload, on the total time the server holds connections open for it: both how many parallel connections and how long each one stays open. That ceiling is the site’s crawl capacity.
How crawl capacity works
Every site starts from the same conservative default ceiling, and when there is crawl demand, Google wanting to crawl more, and the site stays healthy, Google raises the ceiling automatically over time. From there the ceiling moves on two factors. The first is crawl health: stable or improving response times raise the ceiling, and a slowdown, 5xx errors or 429 responses lower it. The second is Google’s own resources, which are finite too and divided across every site on the web.
Google calculates the ceiling per hostname, so a subdomain has a ceiling of its own.
An example of crawl capacity opening up
On a large site I worked on, response time dropped from 500 to 120 milliseconds and daily crawl volume jumped roughly sevenfold. The content did not change in that period. What changed is how many parallel connections Google allowed itself against a server that had stopped straining.
How crawl capacity relates to SEO and GEO
On large sites, crawl capacity is the engineering lever of crawl budget: improving response time raises the ceiling, and when the demand is there, crawl volume rises to match over time. Crawlers like OpenAI’s OAI-SearchBot pull from the same server, and a slow server slows them down too. A page no crawler reached enters neither the index nor the answers.
The difference between crawl capacity and crawl demand
Capacity is how much Google can crawl without hurting the site. Crawl demand is how much it wants to. Capacity is solved with engineering, demand is solved with content. When diagnosing a crawl drop, that is the first split to make: a low ceiling gets fixed in the server, low demand gets fixed in the site.
Questions about Crawl capacity
What lowers crawl capacity in practice?▼
Crawl capacity drops for three reasons: response times stretching, 5xx server errors, and 429 responses telling the bot to slow down. Google reads all of them as signs the site is straining, lowers the ceiling and crawls less.
Where do you see capacity in Search Console?▼
Crawl capacity does not appear as a number in any Google report. Its movement is inferred from the Crawl Stats report, from the average response time chart against requests per day: when response time drops over a sustained period and request volume rises, the ceiling is opening. The one direct signal Google does surface is Hostload exceeded in the URL Inspection tool.
Does a faster server guarantee more crawling?▼
A faster server does not guarantee more crawling: it raises the ceiling, but if crawl demand is low, Google simply will not use the headroom. Google states this outright: even when the capacity limit is not reached, low demand means Google crawls less.
Do all of Google's crawlers share one capacity?▼
The capacity limit is shared across all of Google's crawlers. Each crawler has its own crawl demand, but there is one ceiling, and Google writes that high demand from one crawler can reduce the capacity left for the others.
Can crawl capacity be set manually?▼
Google's crawl capacity cannot be set manually: Google tunes it automatically, and the dedicated tool that existed, the crawl rate limiter in Search Console, could only lower the rate and was retired in January 2024. What remains is slowing Google down through server responses like 503 or 429, not speeding it up.
Bing is different: Bing Webmaster Tools includes Crawl Control, which lets you set an hourly crawl rate in both directions.