Crawl budget

Crawl budget is the number of pages a search engine's crawler will fetch from a given site within a period of time before moving on. Google describes it as combining two things: the crawl rate limit, how fast a bot can request pages without overloading the server, and crawl demand, how worthwhile Google judges it to keep revisiting a site's pages based on their popularity and how often they change.

Crawl budget differs from indexing: a page can be crawled and still not indexed if its content is judged low-value or duplicate, so a healthy crawl budget does not by itself guarantee organic traffic on a search engine results page. Server response speed and page load time directly affect crawl rate, since a slow server causes a crawler to back off and fetch fewer pages per visit.

Google itself has said crawl budget generally is not a concern for most sites — it matters mainly for very large sites, typically many thousands of indexable URLs, or sites generating large numbers of low-value URLs, such as faceted navigation. The common pitfall is assuming every site needs crawl-budget optimization; for most small and medium sites, the more productive fix is improving content quality and site structure, since wasted crawl activity is usually a symptom of that deeper problem, and is unrelated to a site's domain authority score.