robots.txt generator

Generates a robots.txt with disallow and allow rules and a sitemap reference. The critical thing to understand: robots.txt controls crawling, not indexing.

That distinction causes more damage than any other misunderstanding in technical SEO. Disallowing a page stops the crawler fetching it — but if other pages link to it, Google can still index the URL, showing it with no description because it was never allowed to read it. Worse, a page blocked in robots.txt cannot be seen to carry a `noindex` tag, so blocking a page you want removed actively prevents its removal. To keep something out of the index, allow crawling and use `noindex`. To keep something private, use authentication: robots.txt is a public file and a list of the directories you would rather people did not visit is an invitation to the curious. Note also that Google ignores `crawl-delay` entirely.