If you have ever opened your Google Search Console and seen the warning "Indexed, though blocked by robots.txt", you have run into one of the most misunderstood quirks of how Google handles a website. The message sounds contradictory — how can a page be both indexed and blocked? — and most site owners assume it is a bug or a temporary glitch. It is neither. It is the predictable result of using the wrong tool for the wrong job, and the fix is simpler than the warning suggests.
This is a problem we see on almost every legacy website we audit, particularly on sites that were built or maintained by someone who learned SEO from a checklist rather than from how Google actually works. The good news is that once you understand the distinction Google makes between two very different concepts, the fix takes about thirty seconds.
Crawling and Indexing Are Not the Same Thing
The single most common misconception about SEO is treating crawling and indexing as one process. They are two completely separate steps, and they are controlled by completely separate mechanisms.
Crawling is the act of Google's robot fetching the contents of a page so it can read what is on it. Indexing is the act of Google adding a page to its searchable database, where it can appear in search results.
The reason this matters is that you can have one without the other. A page can be indexed without ever being crawled, if Google has discovered it through links from other sites but cannot read its contents. That is exactly what produces the warning we are talking about.
What robots.txt Actually Does
The robots.txt file at the root of your site is a polite request that tells search engine crawlers which paths they are allowed to fetch. A typical line looks like this:
Disallow: /private-page.htmlThat instruction blocks crawling. It does not block indexing. If Google finds a link to that page somewhere on the web — from your own footer, from a competitor's site, from a directory listing — it will add the URL to its index even though it cannot actually read what is on the page. The result is an indexed URL with no content, no description, and no useful snippet for users.
When that happens, Google sends you the warning. It is telling you that you have a page in its search results that it does not understand, and it is asking whether that is what you intended.
The Trap Most People Fall Into
The reason this mistake is so common is that the official-sounding name "robots.txt" suggests it controls what robots do with your pages in general. People assume that disallowing a path keeps that page out of Google entirely. So when they want to hide a thank-you page, a draft article, or a private document, they add a Disallow line and move on.
What actually happens is that the page stays out of the crawl queue but slips into the index through the back door. Users searching for your brand will see the page listed in results, with a generic "No information is available for this page" snippet underneath it. To anyone scanning the search results, this looks suspicious — as if you are hiding something. It also wastes a slot on the search results page that could have shown a more useful URL from your site.
The Right Tool for Each Job
The fix depends on what you actually want to happen. Two different goals require two different tools.
If you want the page to appear normally in Google with proper content, do nothing special. Let Google crawl and index it. Remove any Disallow line that targets it. Most pages on most websites fall into this category — including legal pages like privacy policies and cookie notices that many owners reflexively try to hide. Those pages are fine to index. They take nothing away from your SEO and they signal to Google that you run a legitimate business.
If you want the page completely out of Google's search results, you need to do the opposite of what robots.txt does. You need to let Google crawl the page, then add a tag inside the page that tells Google not to index it. The tag goes inside the <head> section:
<meta name="robots" content="noindex,follow" />That single line is far more effective than any robots.txt rule. Google crawls the page, reads the tag, and removes the URL from its index. The "follow" part tells Google it can still follow the links inside the page, which preserves the flow of authority through your site.
A Quick Real-World Example
Imagine you have a private internal staging page at /staging-preview.html that you do not want appearing in search. Most site owners would write this in robots.txt:
User-agent: *
Disallow: /staging-preview.htmlThis is the trap. If anyone links to the staging page from anywhere — even accidentally, even from a forgotten internal document — Google will index the URL without content. The proper approach has two parts. First, in robots.txt, do not block it. Second, on the staging page itself, add the noindex tag in the head:
<meta name="robots" content="noindex,follow" />Google will crawl the page, see the tag, and keep the URL out of the index entirely. No warning, no orphaned listing, no awkward snippet.
When robots.txt Is Actually Useful
This is not a piece against robots.txt. It remains a perfectly good tool for a specific job: telling crawlers to skip parts of your site that are not worth crawling at all. Large directories of duplicate URLs, faceted search results, admin paths, server-rendered preview tokens — these belong in robots.txt because there is nothing on those pages worth indexing in the first place. Saving the crawl budget for your real content is a legitimate use case.
What robots.txt is not is a privacy mechanism, an access control, or a way to keep pages out of search results. Anyone can read your robots.txt file directly, so listing private URLs there actually makes them easier to discover. And as we have just seen, blocking the path does not stop the URL from being indexed when it is linked from elsewhere.
How to Fix It on Your Own Site
If you are reading this because Search Console sent you the warning, the steps are quick:
- Open your robots.txt file at
yourdomain.com/robots.txtand identify theDisallowlines for the affected pages. - Decide what you actually want. Should the page appear in Google? Remove the
Disallowline. Should it not? Remove the line and add the noindex meta tag to the page. - Deploy the change and wait for Google to refresh its cached copy of your robots.txt. This usually takes a few hours to a day.
- Verify in Search Console by using the URL Inspection tool on the affected page. The warning should disappear once Google re-crawls the page with the new instructions.
If you are not sure whether your site has this problem, the fastest way to check is to open Search Console, go to the Pages report, and look for the "Indexed, though blocked by robots.txt" row. If it is there, those pages are the ones to start with.
The Bigger Lesson
Most SEO mistakes happen because the tools are named in a way that suggests they do something they do not. robots.txt sounds authoritative, but it only controls the politest part of crawling. Meta tags inside the page itself are what actually govern indexing. Sitemaps suggest pages to Google but do not guarantee anything. Canonical tags hint at preferred versions but Google still chooses. Every one of these tools has a specific, narrow job, and the trouble starts when they are pressed into service for something else.
The fix in every case is the same: figure out what behavior you actually want, then pick the tool that controls that specific behavior. With robots.txt and indexing, the right answer is almost always to leave robots.txt alone and use the noindex meta tag — or, more often than you might expect, to do nothing at all and let Google do its job.