Generate an XML sitemap that never lists a blocked page
Crawl your site to build a valid XML sitemap, or paste your URLs. It excludes anything your robots.txt blocks, validates and dedupes, and splits past 50,000 URLs into a sitemap index. The crawl runs on our server; the XML is built in your browser.
A sitemap is how you hand a search engine a clean list of the pages you want found. It looks like a trivial file, and that is exactly why it goes wrong: it is capped at 50,000 URLs and 50MB, every URL has to be absolute and properly escaped, a sitemap may only list URLs from its own host, and the mistake almost nobody catches, a sitemap should never contain a URL that robots.txt blocks from crawling. Point a crawler at a sitemapped page it is not allowed to crawl and you have handed it a contradiction that quietly wastes crawl budget and keeps the page out of the index.
What a sitemap is actually for
A sitemap does not make Google index a page, and it is not a ranking factor. What it does is help discovery: on a large site, a new site, or one with pages that are not well linked internally, it makes sure the crawler knows every URL exists. The single most useful field is lastmod, an honest last-modified date, which tells crawlers what has changed since they last visited so they recrawl the right pages rather than everything.
The check no other generator does
The rule people miss is that the sitemap and robots.txt have to agree. A URL listed for indexing but blocked from crawling is a contradiction. When you crawl a site here, the tool reads that site's robots.txt and drops every URL it blocks, using the same matching rules Google uses, so the sitemap you get is consistent by construction. It also tells you how many URLs it excluded, so nothing is silently dropped.
Valid, and correct at scale
Every URL is validated and deduplicated, non-URLs and mixed domains are flagged, and the output is escaped XML that passes validation. Past the 50,000-URL limit the tool does what the protocol requires and splits the file into numbered sitemaps under a single sitemap index, rather than the one oversized file most generators hand back that a crawler will simply reject.
How to use it
- 1
Crawl or paste
Enter your domain to crawl internal links from the homepage, or switch to Paste URLs and drop in a list you already have. Either way the tool takes it from there.
- 2
Set the optional tags
Add a last-modified date, change frequency and priority if you want them. Google largely ignores changefreq and priority and pays attention to lastmod, so an honest date is the one worth setting.
- 3
Review the warnings
Check the notes: skipped invalid lines, removed duplicates, http URLs to fix, or a split into an index. Fix anything flagged before you ship.
- 4
Upload and submit
Download the file, or the index and its parts, upload them to your site root, add a Sitemap line to robots.txt, and submit the sitemap in Google Search Console.
Common mistakes
Listing URLs that robots.txt blocks
The contradiction that wastes crawl budget: a URL in the sitemap that robots.txt disallows. The crawler cannot fetch it, so listing it does nothing but confuse. Crawling here removes them automatically; if you paste URLs, check them against your robots.txt first.
Including non-canonical or redirecting URLs
A sitemap should list the final, canonical, 200-status URL for each page, not a redirect, a parameter variant, or a noindex page. Duplicates and redirects dilute the signal. Keep it to the one URL you actually want indexed.
Shipping one giant file over the limit
A single sitemap tops out at 50,000 URLs and 50MB uncompressed. Past that, crawlers reject the file. Split into multiple sitemaps under an index, which this tool does for you automatically.
Letting it go stale
A sitemap with stale lastmod dates, or one that never updates when you publish, teaches crawlers to trust it less. Regenerate it when your content changes, or generate it dynamically, and keep the dates honest.
Questions
- Does a sitemap help my ranking?
- Not directly. It helps search engines discover and recrawl your pages, which matters most on large sites, new sites, or sites with weak internal linking. It does not push a page up the results; it makes sure the page is found in the first place.
- Where do I put the sitemap?
- Upload it to your site, commonly at yoursite.com/sitemap.xml, add a Sitemap: line pointing to it in your robots.txt, and submit it in Google Search Console. Each host needs its own sitemap.
- How is crawling different from pasting?
- Crawl visits your homepage and follows internal links to discover pages automatically, up to 60, and drops anything your robots.txt blocks. Paste is for when you already have a URL list, or a larger site where you want full control. Both produce the same validated XML.
- Why did some URLs get excluded?
- The crawler skips pages your robots.txt disallows, since they should not be in a sitemap, along with assets like images and scripts, URLs with query strings, and anything off your domain. What is left is the clean, indexable list a sitemap is meant to contain.
- What are changefreq and priority for?
- They are hints about how often a page changes and how important it is relative to your other pages. In practice Google has said it mostly ignores both. lastmod, an accurate last-modified date, is the field that still carries weight, so it is the one to get right.
Need this done properly, at scale?
The tool handles the one-off. When it's a system you're building, that's the paid version of the job, and we do that too.
SEO Services