Crawl Budget

Crawl budget is the number of pages a search engine is willing and able to crawl on a given site within a given timeframe.

Crawl budget is mostly a concern for large sites (tens of thousands of pages or more) or sites generating pages dynamically — for a small site with a few hundred well-structured pages, crawl budget is rarely the bottleneck limiting rankings. Google itself has said crawl budget isn't something most sites need to worry about, but it becomes real for large ecommerce catalogs, faceted navigation systems, or sites with a lot of thin, low-value URLs eating into the budget that could otherwise go to important pages.

Wasting crawl budget on low-value pages (auth-gated dashboards, infinite filter combinations, duplicate parameter URLs) is the usual practical issue — blocking those via robots.txt or noindex redirects crawler attention toward pages that actually matter.