Robots.txt Generator

What is robots.txt?

robots.txt is a plain text file placed at the root of a website (e.g. example.com/robots.txt) that tells well-behaved search engine crawlers which parts of the site they may or may not crawl, and optionally points them to your sitemap.

How to use it

1. Choose "Allow all" if you want every page crawled — the right choice for most public sites.
2. Or choose "Custom rules" to block specific paths, such as an admin area or checkout pages, and optionally re-allow a sub-path inside a blocked one.
3. Add your sitemap URL, then copy the result and save it as a file named exactly robots.txt in your site's root folder.

robots.txt is a request, not a security barrier — well-behaved crawlers like Googlebot respect it, but it doesn't prevent a page from being accessed directly, and it shouldn't be used to hide sensitive content (use proper authentication for that). Blocking a page in robots.txt also doesn't guarantee it won't appear in search results if other sites link to it; for that, use a noindex meta tag on the page itself instead. Getting robots.txt wrong — for example, accidentally disallowing / — can block your entire site from search engines, so it's worth double-checking the generated rules before publishing.