"Indexed, Though Blocked by robots.txt": What It Means and the Right Fix
"Indexed, though blocked by robots.txt" means Google put a URL in its index even though your robots.txt tells it not to crawl that URL, usually because other pages link to it. The fix depends on one question: should this page be in Google or not? If yes, remove the robots.txt rule; if no, remove the rule anyway and add noindex, because robots.txt cannot keep a page out of the index.

What the warning actually means
Two different things are happening at once, and the warning exists because they contradict each other:
- Crawling is Googlebot downloading the page. Your robots.txt says "don't."
- Indexing is Google storing the URL so it can show up in results. Nothing told Google "don't" about that.
Google's own Page indexing report help says the URL "was indexed despite being blocked" and states plainly that robots.txt "is not the correct mechanism to avoid being indexed." The robots.txt introduction repeats it: robots.txt is for managing crawler traffic, and a blocked URL "can still appear in search results," just without a description.
So Google found links pointing at the URL (internal links, your sitemap, someone else's site), could not read the page, and indexed the bare address anyway. That's why these results often show a title guessed from anchor text and a line like "No information is available for this page."
Is it a problem? Usually it's a mismatch, not a penalty
The warning is a yellow flag, not an error that hurts the rest of your site. It tells you that your instructions and your intent don't line up. One of two things is true:
- You blocked a page you actually want in search (a mistake in robots.txt, a leftover staging rule, an over-broad wildcard). This one costs you traffic.
- You blocked a page you don't want in search, and it got in anyway. This one costs you a messy result with no snippet, and possibly an exposed URL you'd rather hide.
Both get fixed in Search Console within the same afternoon. Which one you're in decides the fix.
The decision tree: two questions, three fixes
| Question 1: Should this URL appear in Google? | Question 2 | Fix |
|---|---|---|
| Yes (a product, post, category, landing page) | - | Fix A: remove or narrow the Disallow rule, then request indexing |
| No, but it's harmless if found (filters, internal search, cart, tag archives) | Do you need to save crawl capacity on a very large site? | Usually Fix B: allow crawling and add noindex. On a huge site with millions of parameter URLs, keeping the Disallow and accepting the warning can be a deliberate choice |
| No, and it must not be seen (private files, client data, staging) | - | Fix C: password-protect or delete it (404/410), then use the Removals tool for speed |
A rule of thumb that resolves most cases: robots.txt controls crawling, noindex controls indexing, a password controls access. Pick the tool that matches the goal.
Step 0: find out which URLs and which rule
- In Search Console, open Indexing → Pages, scroll to "Why pages aren't indexed" or the improvement section, and click Indexed, though blocked by robots.txt. Export the example list.
- Group the URLs by pattern. Typical clusters:
/?s=search pages,?orderby=or?filter_parameters,/cart/,/wp-admin/endpoints,/tag/, PDFs in an upload folder. - Run one URL from each cluster through URL Inspection. It shows "Crawl allowed? No: blocked by robots.txt," which confirms the cause.
- Open your live file at
yourdomain.com/robots.txtand find the line that matches. Remember how Google resolves conflicts: per the robots.txt specification, the most specific (longest) matching rule wins, and when rules conflict Google uses the least restrictive one. A shortDisallow: /*?can quietly block thousands of URLs you never thought about.
Decide per cluster, not per URL. You'll usually end up with two or three decisions, not two hundred.
Fix A: the page should be indexed
- Edit robots.txt: delete the rule, or narrow it. For example, replace
Disallow: /blog(which also blocks/blog/and every post) with the exact path you meant, such asDisallow: /blog-drafts/. - Check that nothing else blocks the page: no
noindexmeta tag, noX-Robots-Tagheader, a 200 status code. - Google generally caches robots.txt for up to 24 hours (source). If it's urgent, open Settings → robots.txt report and request a recrawl of the file; the report's help page describes that option for emergencies.
- Run URL Inspection → Test live URL on a sample page, then Request indexing for the most important ones.
- Back in the Pages report, click Validate fix on the issue so Google rechecks the whole group.
Once Google can crawl the page, it gets a real title and snippet instead of the empty placeholder.
Fix B: the page should stay out, and it's not secret
This is the counterintuitive one. To get a page out of the index, you have to let Google in. Google's noindex documentation is explicit: for noindex to work, the page "must not be blocked by a robots.txt file," otherwise the crawler never sees the rule.
- Add noindex to the page. In HTML:
<meta name="robots" content="noindex">. For PDFs, images and other non-HTML files, send the HTTP headerX-Robots-Tag: noindexfrom the server. - Remove the matching Disallow line so Googlebot can fetch the page and read the tag.
- Remove these URLs from your XML sitemap. A sitemap says "please index this," which contradicts noindex.
- Wait for the recrawl. As Google sees the tag, the URLs move to "Excluded by 'noindex' tag," which is the outcome you want.
- Optional, after they've dropped out: if the pages waste a lot of crawling, you can put the Disallow back. Google already knows they're noindexed, but new URLs of the same pattern could be indexed blind again, so do this only on very large sites.
Don't try the shortcut of writing Noindex: inside robots.txt. Google retired support for that unofficial rule on September 1, 2019.
Fix C: the page must not be seen at all
robots.txt is a public file, so a Disallow line pointing at /private-reports/ is effectively a signpost. For anything confidential:
- Put it behind a login or server-side password (HTTP authentication), or delete it so the server returns 404 or 410.
- For speed, file a request in the Removals tool. Per Google's Removals help, a successful request "lasts only about six months," so it hides the URL while the permanent fix (deletion, password or noindex) takes effect. It is not a fix on its own.
- If the URL is still blocked in robots.txt, unblock it, so Google can recrawl, see the 404 or the login, and drop it for good.
Common causes by platform
| Where | Typical cause | What to check |
|---|---|---|
| WordPress | A hand-edited or plugin-generated robots.txt blocking /?s=, /tag/, /wp-content/uploads/ or a whole section | SEO plugin's file editor, or a physical robots.txt in the site root that overrides the virtual one |
| WordPress, old sites | A leftover Disallow: / from development | Since WordPress 5.3, "Discourage search engines" outputs a noindex meta tag instead of a robots.txt block (core announcement). A static Disallow-all file is usually manual |
| Online stores | Faceted filters and sort parameters blocked with wildcards | Whether any filter combination is a page you want ranking (for example a "red dresses" category) |
| Any site | Internal links or the sitemap pointing at blocked URLs | Remove blocked URLs from the sitemap; fix or nofollow internal links to pages you don't want found |
| Staging copies | A staging subdomain blocked only by robots.txt | Add HTTP authentication; robots.txt alone leaks it |
Worked example (illustrative)
A small WordPress blog sees 140 URLs under "Indexed, though blocked by robots.txt." The export splits into three clusters:
- 96 internal search URLs (
/?s=best+mixer), blocked byDisallow: /?s=. Decision: should not rank, not secret → Fix B. The owner enables noindex on search results in the SEO plugin and removes the Disallow line. - 38 tag archives, blocked by
Disallow: /tag/. Decision: same, Fix B, and removes tags from the sitemap. - 6 recipe posts with a
/recipes-old/slug caught byDisallow: /recipes. Decision: these should rank → Fix A. The rule becomesDisallow: /recipes-drafts/, the robots.txt recrawl is requested, and the 6 posts get "Request indexing."
The numbers are made up for the example; the pattern is typical. Three rules changed, one validation started, done.
Checklist
- Exported the affected URLs and grouped them by pattern
- For each group, answered "should this be in Google?"
- Pages that should rank: Disallow removed, no noindex, 200 status, indexing requested
- Pages that shouldn't rank: noindex added, Disallow removed, out of the sitemap
- Private pages: password or 404/410, Removals request filed
- robots.txt recrawl requested in the robots.txt report if urgent
- "Validate fix" clicked in the Pages report
If you're also blocking AI crawlers in the same file, double-check that those rules target only the bots you meant; our guide to blocking AI crawlers and the free AI crawler checker show what your current robots.txt allows.
What to do next
Fix the warning, then look at the next indexing status in the report; the logic is different for each. See Crawled, currently not indexed and Discovered, currently not indexed, and the short definition of noindex.
Robots rules are only one part of how a site looks from the outside. Run your domain through Satelo's free check to see its Quality Score with the evidence behind each number, mostly from open data, in about 30 seconds.
FAQ
What does "Indexed, though blocked by robots.txt" mean?
Google indexed the URL without crawling it, because robots.txt blocks crawling but other pages or your sitemap link to it. Google stores the bare address and may show it in results without a description.
Is "Indexed, though blocked by robots.txt" bad for SEO?
It doesn't penalize the rest of your site. It signals a mismatch: either a page you want ranking is blocked, or a page you want hidden got indexed anyway. Fix whichever one it is.
Does robots.txt stop a page from being indexed?
No. Google's documentation says robots.txt is not a mechanism for keeping a page out of Google. Use a noindex meta tag or X-Robots-Tag header, and make sure the page is crawlable so Google can see it.
How do I remove a page that is indexed but blocked by robots.txt?
Remove the Disallow rule, add noindex, take the URL out of your sitemap and wait for a recrawl. For confidential content, password-protect or delete it and use the Removals tool, which hides a URL for about six months.
Can I put noindex in robots.txt?
Not for Google. It stopped supporting the unofficial noindex rule in robots.txt on September 1, 2019. Use the robots meta tag or the X-Robots-Tag HTTP header instead.
How long does it take for the warning to go away?
Google generally caches robots.txt for up to 24 hours, then the affected URLs need to be recrawled. You can request a robots.txt recrawl, request indexing for key URLs and click Validate fix to speed things up; there's no guaranteed timeline.
Should I worry about blocked filter or search URLs?
Usually not much. Many sites see this for internal search and filter parameters. If you don't want them in Google, switching from Disallow to noindex is the cleaner fix; very large sites sometimes keep the Disallow on purpose to save crawling.
Read next




Check your site's Quality Score
Free, no sign-up: a 0–100 score from mostly open data. Then add your site to Satelo and get partner articles from sites of your level.
Enter a domain like example.com
Add my site