Build a robots.txt without hiding your site by mistake
Generate a correct robots.txt from a few safe choices, with plain-language explanations of what each rule does. Defaults to allowing everything. Nothing leaves your browser.
A robots.txt is four lines of plain text with the power to remove your entire site from Google. The single most common catastrophic SEO mistake is a robots.txt copied from a staging server, where blocking all crawlers is correct, and shipped to production, where it is a disaster. The file looks harmless, nobody reviews it, and a site can sit invisible in search for months before anyone connects the missing traffic to one stray Disallow line.
What robots.txt actually does
It tells well-behaved crawlers which parts of your site they may request. It is a request, not a lock: it keeps search engines out of the folders you name, but it is not security, and it does not hide a page that is linked from elsewhere. Its real job is to stop crawlers wasting time on pages that should not be indexed, like an admin area, a cart, or an internal search results page, so they spend their limited crawl budget on the pages that matter.
Why the default should be to allow everything
Most sites need a robots.txt that blocks almost nothing. The instinct to lock things down is usually wrong: blocking a folder in robots.txt does not remove those pages from Google if they are linked to, it just stops Google reading them, which can leave a worse result with no description. The safe default, and this tool's default, is to allow all crawlers and only block the handful of paths that genuinely should never be indexed.
The one line that hides everything
Disallow: / under User-agent: * blocks your entire site from every crawler. It is exactly right on a staging site and catastrophic on a live one, and the two files look almost identical, which is how the mistake happens. This tool makes that option explicit, labels it as staging-only, and warns you in red when you select it, so the dangerous choice is never the one you make by accident.
How to use it
- 1
Choose a starting point
Allow everything is the safe default for a live site. Allow-but-block lets you keep specific folders out. Block everything is for staging only, and the tool warns you.
- 2
Name any paths to block
Admin areas, carts, internal search. One per line. The tool adds the leading slash if you forget it.
- 3
Add your sitemap URL
Pointing crawlers at your sitemap from robots.txt helps them find every page. Paste the full URL to your sitemap.xml.
- 4
Save it as robots.txt at your root
The file must live at yoursite.com/robots.txt, exactly. Not in a folder. Copy the output and put it there.
Common mistakes
Shipping the staging robots.txt to production
The number one robots.txt disaster. A staging file blocks all crawlers; the same file on a live site hides it from Google. Check your live robots.txt after every deploy.
Using it to hide private pages
robots.txt is public and is not security. Anyone can read yoursite.com/robots.txt, and listing a secret path there advertises it. Protect private pages with authentication, not robots rules.
Blocking CSS and JavaScript
Old advice said to block these; modern Google needs them to render and judge your page. Blocking them can hurt your ranking. Leave your assets crawlable.
Forgetting the sitemap line
It is not required, but adding Sitemap: to robots.txt is a free way to make sure crawlers find your sitemap. There is no reason to leave it out.
Questions
- Where does robots.txt go?
- At the exact root of your domain: yoursite.com/robots.txt. It does not work in a subfolder, and each subdomain needs its own. This tool's output is the file contents; you place the file.
- Does robots.txt remove a page from Google?
- No, and this trips people up. Blocking a page stops Google reading it, but if the page is linked from elsewhere it can still appear, with no description. To remove a page, use a noindex meta tag or password-protect it.
- Should I block AI crawlers?
- Your call. Blocking GPTBot, ClaudeBot and similar stops your content being used for AI training, and does not affect Google Search. The tool has a one-click toggle if you want it. Many sites now do.
- What is crawl-delay?
- It asks crawlers to wait a set number of seconds between requests, to reduce server load. Google ignores it, but some other crawlers respect it. Most sites do not need it.
- How do I test my robots.txt?
- Google Search Console has a robots.txt report that shows what Google fetched and flags errors. Check it after you publish, and especially after a site migration.
Related tools
Need this done properly, at scale?
The tool handles the one-off. When it's a system you're building, that's the paid version of the job, and we do that too.
SEO Services