noindex

noindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in search results. The crawler must be able to fetch the page to see the rule, so a page blocked in robots.txt can stay indexed despite a noindex tag.

Updated: 2026-10-16 · Glossary: Crawling and indexing

How to set it

• In HTML: <meta name="robots" content="noindex"> in the page's head.
• In an HTTP header (works for PDFs and other files): X-Robots-Tag: noindex.

Google's documentation on blocking indexing explains both methods and warns that if robots.txt blocks the URL, Googlebot never sees the noindex rule, so the page can still appear in results (usually without a snippet) when other pages link to it. Yandex's help documents the same directive for its robot.

When small sites should use it

• Thin archive pages: tag pages with one post, internal search results, empty category pages.
• Thank-you, login, cart and account pages.
• Staging copies, if they cannot be password-protected.

Do not noindex pages you want to rank, and check after theme or SEO-plugin changes: an accidental site-wide noindex is a classic reason for traffic to vanish. If pages are indexed but not shown, look at “Crawled – currently not indexed” instead.

How Satelo relates

Partner articles in Satelo are published as normal, indexable posts on the host site, labelled as partner content, with rel="sponsored" links; the host keeps control over its own SEO settings. On satelo.io itself, the dashboard and API endpoints are kept out of search while content pages stay open to search and AI crawlers. Related: crawl budget.

Related terms

• Crawl budget: Crawl budget is the number of URLs a search engine crawler can and wants to fetch from a site in a given period.
• IndexNow: IndexNow is an open protocol that lets a website notify participating search engines the moment a URL is added, updated or deleted, instead of waiting for the next crawl.
• llms.txt: llms.txt is a proposed standard for a Markdown file placed at the root of a website (/llms.txt) that gives large language models and AI agents a short, curated overview of the site: a title, a summary, and lists of the most useful links.

← All glossary terms