A disallow rule tells a crawler not to fetch a page. It does not tell a search engine not to list it, and those are different instructions — which is why a page blocked in robots.txt can still appear in results, usually with no description underneath.
The instruction that removes a page from an index is a noindex directive on the page itself. That creates the contradiction at the centre of this topic: a crawler has to fetch a page to see that directive, and a blocked page is never fetched.
Which instruction does what?
Four mechanisms, each addressing a different question.
| Instruction | What it controls |
|---|---|
| robots.txt disallow | whether the page is fetched |
| noindex | whether the page is listed |
| canonical | which URL of several is preferred |
| redirect | where the URL now lives |
To remove a page from search, allow it to be crawled and give it a noindex directive. Once it has dropped out, blocking it in robots.txt is safe — but doing both at once leaves it listed indefinitely.
The reason a blocked page gets listed at all is links. Other pages pointing at it tell the engine the URL exists and roughly what it is about, which is enough for a bare entry.
Is a canonical tag obeyed?
It is a strong hint and not a command. A search engine considers the declared canonical alongside its own signals — internal links, sitemap entries, redirects, content similarity — and can choose a different URL.
When it disagrees, the usual cause is contradiction. A page declaring one canonical while every internal link points at another, or a sitemap listing the version the canonical points away from, gives conflicting evidence and the tag loses.
Two rules avoid nearly all of it: make the canonical an absolute URL, and give every page a self-referencing canonical unless it genuinely is a duplicate. Self-reference is what stops parameters and tracking suffixes creating alternate versions.
Why do hreflang tags fail?
Missing return links. A set of language alternates has to be mutually complete — every page in the set names every other page including itself — and one page missing from one set invalidates the relationship.
The other frequent error is the code itself. The value is a language code and an optional region code, and the region code comes from a standard list that does not always match intuition: the United Kingdom is GB, not UK.
A fallback entry is worth adding for visitors whose language is not in the set. It names the page to use when nothing else matches, and without it those visitors get whichever version the engine guesses.
What belongs in a sitemap?
Canonical, indexable URLs and nothing else. A sitemap listing redirects, blocked pages or non-canonical duplicates sends the opposite signal to the tags on those pages.
There is a size limit — fifty thousand URLs and fifty megabytes uncompressed per file — and larger sites split across several files with an index pointing at them. That is a structural limit rather than a recommendation.
The last-modified date is used when it is credible. A site stamping every URL with today’s date on every build is providing no information, and the field stops being trusted, so it is better left out than fabricated.
Which redirect should you use?
A permanent one for a move that is permanent, and a temporary one for anything you intend to reverse. The difference matters because a permanent redirect transfers the ranking signals of the old URL and is cached hard by browsers.
That caching is the reason to be careful. A permanent redirect set up by mistake stays in visitors’ browsers after it is removed from the server, and there is no way to reach in and clear it.
There are variants of both that preserve the request method, which matters for anything other than a plain page view. For ordinary page moves the classic pair is what everything expects.
What does a crawl budget actually constrain?
How much of a large site gets fetched in a given period. It is a real constraint on sites with hundreds of thousands of URLs and close to irrelevant below that, which is the opposite of how often it gets discussed.
Where it bites, the cause is usually URLs that should not exist: faceted filters generating endless combinations, session parameters, or a calendar with no end. Blocking those patterns is what robots.txt is genuinely good at, because the goal is to prevent fetching rather than to prevent listing.
For an ordinary site the useful version of this concern is much simpler — make sure the pages you care about are linked from somewhere and present in the sitemap.
Questions people ask
Does robots.txt keep a page private? No. It is a public file listing exactly what you did not want fetched.
Can I noindex from robots.txt? That was never reliably supported and is now explicitly unsupported. Use the page directive.
Do meta descriptions affect ranking? No. They affect whether a listing is clicked, and are often rewritten anyway.
How long does removal take? Days to weeks after recrawl. There are faster tools for urgent removals.
Write the head tags deliberately. The meta tag generator covers the basics, the canonical tag generator and hreflang tag generator handle the duplicate and language questions, and the robots.txt generator, sitemap XML generator and htaccess redirect generator cover the site-level files.