Robots.txt Generator
Build a robots.txt, then test it the way Google really reads it
Generate a correct robots.txt, then test any path against it using Google's real matching rules, the tester Google removed from Search Console. Check it per bot, lint it for the mistakes that quietly cost traffic, and audit any live site against its own sitemap. The builder and tester run in your browser; only the live-site audit fetches the URL you enter.
- 100% free
- No sign-up
- Nothing uploaded
- Runs in your browser
A robots.txt is a few lines of plain text with the power to remove your entire site from Google. The classic disaster is a file copied from a staging server, where blocking every crawler is correct, and shipped to production, where it is not. And since Google retired the robots.txt tester from Search Console, there is no longer an official way to ask a simple question: is this path actually blocked? Most free testers that remain do a naive prefix match, which gives the wrong answer on exactly the tricky files, the ones with an Allow exception, a wildcard, or a rule for a specific bot.
What it does, and why it's built this way
What robots.txt actually does
It tells well-behaved crawlers which parts of your site they may request. It is a request, not a lock: it keeps search engines out of the folders you name, but it is not security, and it does not remove a page that is linked from elsewhere. Its real job is to stop crawlers spending time on pages that should not be indexed, an admin area, a cart, an internal search page, so their limited crawl budget goes to the pages that matter.
Testing, not guessing
This tool implements the matching algorithm Google documents, not a rough approximation. It picks the most specific user-agent group, then within it the longest matching rule, honours the * wildcard and the $ end anchor, and lets an Allow override a Disallow the way Google does. You can test a path as Googlebot, Bingbot or an AI crawler and see exactly which rule decided the outcome, which is the answer the retired Search Console tester used to give.
The mistakes that cost traffic
A built-in linter flags the errors people ship without noticing: hiding the whole site, blocking the CSS and JavaScript Google needs to render a page, trusting a Crawl-delay that Googlebot ignores, or confusing crawl control with index control. Point it at a live domain and it pulls the real robots.txt, runs the same checks, and cross-references the sitemap to find any URL that is listed for indexing but blocked from crawling, a contradiction that silently keeps pages out of Google.
How to use it
- 1
Choose a starting point
Allow everything is the safe default for a live site. Allow but block keeps specific folders out. Block everything is for staging only, and the tool warns you.
- 2
Block paths, and carve exceptions
Name the paths to block, one per line, and add Allow exceptions where you need to let one file back out of a blocked folder, like admin-ajax.php inside a blocked wp-admin. Platform presets fill the right paths for WordPress, Shopify and Next.js.
- 3
Test a path as a specific bot
Type a path and pick a crawler. The tester tells you allowed or blocked and shows the exact rule that matched, using Google's real precedence.
- 4
Audit your live site, then save at the root
Enter your domain to pull and lint the real robots.txt and cross-check its sitemap. When the file is right, save it as robots.txt at the exact root of your domain.
Common mistakes
Shipping the staging robots.txt to production
The number one robots.txt disaster. A staging file blocks all crawlers; the same file on a live site hides it from Google. The linter flags a whole-site block, and the live audit lets you confirm production after every deploy.
Using it to hide private pages
robots.txt is public and is not security. Anyone can read yoursite.com/robots.txt, and listing a secret path there advertises it. Protect private pages with authentication, not robots rules.
Blocking CSS and JavaScript
Old advice said to block these; modern Google needs them to render and judge your page. The linter warns when a rule looks like it blocks assets or a framework folder such as _next, so leave those crawlable.
Trusting Crawl-delay for Google
Googlebot ignores Crawl-delay entirely; only some other crawlers honour it. To slow Google down, set the crawl rate in Search Console. The linter points this out so you do not rely on a line that does nothing.
Frequently asked questions
- Where does robots.txt go?
- At the exact root of your domain: yoursite.com/robots.txt. It does not work in a subfolder, and each subdomain needs its own. This tool's output is the file contents; you place the file.
- Does robots.txt remove a page from Google?
- No, and this trips people up. Blocking a page stops Google reading it, but if the page is linked from elsewhere it can still appear, with no description. To remove a page, allow crawling and add a noindex meta tag, or password-protect it.
- How do I test my robots.txt now that Search Console removed the tester?
- Here. Type a path, pick a crawler, and the tool evaluates it with Google's real rules, wildcards, the $ anchor and Allow overrides included, and shows which rule matched. You can also audit a live domain's file directly.
- Should I block AI crawlers?
- Your call. Blocking them stops your content being used for AI training or answers, and none of it affects your Google Search ranking. The tool groups bots into AI training, AI search and SEO scrapers so you can block a whole category with one toggle.
- Does it support Allow rules and wildcards?
- Yes. You can add Allow exceptions to carve a path out of a blocked folder, and the tester understands * wildcards and the $ end anchor, so it matches the behaviour of real crawlers rather than a simple prefix check.
Need this done properly, at scale?
The tool handles the one-off. When it's a system you're building, that's the paid version of the job, and we do that too.
SEO Services