robots.txt Generator

Build a robots.txt from presets or custom disallow rules. Free, runs entirely in your browser — no signup, and nothing you paste in is uploaded.

robots.txt controls crawling, not indexing, and conflating the two causes the most common mistake in SEO. A disallowed URL can still appear in search results if other pages link to it — the crawler simply never fetches it, so the listing has no description. To keep a page out of the index you need a noindex tag, which requires the crawler to be allowed to fetch it.

Two more worth knowing: the file only applies to its own origin, so a subdomain needs its own, and it is public by definition. Listing a secret path here is announcing it to everyone who types the filename, which is the opposite of hiding it.