Crawl budget

Crawl budget is the number of URLs a search engine crawler can and wants to fetch from a site in a given period. Google describes it as a combination of crawl capacity (how much your server can handle) and crawl demand (how much Google wants to crawl, based on popularity and freshness).

Updated: 2026-10-16 · Glossary: Crawling and indexing

Who should worry

Google's crawl budget guide says it is intended mainly for large sites (around a million or more unique pages changing weekly), medium or larger sites (10,000+ pages) whose content changes daily, and sites with a large share of URLs reported as “Discovered – currently not indexed”. For most small sites, Google states that keeping the sitemap up to date and checking index coverage is enough.

Why it matters for small sites anyway

Small sites rarely run out of crawl capacity, but they often have low crawl demand: few links and little popularity mean the crawler visits rarely. The symptoms look like a budget problem (new pages not indexed for weeks) but the cause is weak signals. Our notes from a new site are in “Crawled – currently not indexed”. Useful steps:

• Do not waste crawls: avoid endless parameter URLs, keep thin tag archives out of the index with noindex, fix redirect chains.
• Make the server fast and stable; errors reduce crawl capacity.
• Submit a clean XML sitemap and, for Bing and Yandex, notify changes with IndexNow.
• Raise demand with real mentions on related sites.

How Satelo relates

Satelo does not control crawling. Its Quality Score looks at content signals a crawler also sees (sitemap size, freshness, response time), and a labelled partner article on a related site in the network is one more place where crawlers and readers can discover your site. Satelo makes no indexing or ranking promises.

Related terms

• noindex: noindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in search results.
• IndexNow: IndexNow is an open protocol that lets a website notify participating search engines the moment a URL is added, updated or deleted, instead of waiting for the next crawl.
• Satelo Quality Score: The Satelo Quality Score is a 0–100 rating of a website built mostly from open data: popularity lists, open link graphs, domain age, plus a crawl of the site's content and technical health.
• llms.txt: llms.txt is a proposed standard for a Markdown file placed at the root of a website (/llms.txt) that gives large language models and AI agents a short, curated overview of the site: a title, a summary, and lists of the most useful links.

← All glossary terms