XML Sitemap & URL Architecture Detector

Sitemap Detector & XML Sitemap Finder

An enterprise-grade, ultra-fast XML sitemap discovery, validation, decompressor, and URL extraction engine for BrowserMesh.

Discovers, verifies, decompresses, and categorizes XML sitemaps, nested sitemap indexes, RSS/Atom feeds, and gzipped (.xml.gz) files from any website without brute-force website crawling or heavy page downloads.

The premier high-speed, free alternative to Apify Sitemap Detector ($49+/mo), Screaming Frog (£209/yr), Octoparse, and brittle regex-based sitemap scrapers.

🎯 Target Audience

• Technical SEO Specialists & Web Auditors: Rapidly audit XML sitemap health, indexation readiness, crawl budget efficiency, and canonical discovery paths across hundreds of client domains.
• Content Marketers & Digital Strategists: Harvest high-value blog, article, landing page, and category URLs in seconds to evaluate competitor publishing velocity.
• E-Commerce & Retail Analysts: Uncover multi-partition Shopify, WooCommerce, Magento 2, and BigCommerce product catalog sitemaps (sitemap_products_1.xml) to calculate competitor SKU inventory depth.
• Data Engineers & Web Scrapers: Use sitemaps as clean, authoritative URL feeds to bootstrap high-throughput downstream scraping workflows without writing custom crawling spiders.
• Growth Marketers & B2B Researchers: Discover newly launched product lines, localized international subdirectories, and media assets using sitemap <lastmod> timestamps.

🚀 Key Features

1. 5-Stage Multi-Engine Sitemap Discovery Pipeline

Discovers sitemaps across every possible publication vector:

• IETF RFC 9309 Robots.txt Engine: Case-insensitively parses Sitemap: directives, strips inline comments, and automatically resolves relative paths to secure absolute URLs.
• HTML <head> & DOM Link Inspection: Detects <link rel="sitemap"> tags, alternate feed links, and footer navigation anchors.
• HTTP Header Engine: Inspects RFC 8288 and RFC 5988 Link: <...>; rel="sitemap" response headers.
• Curated Dictionary Probing: Fast mode checks the top 20 universal and CMS paths (WordPress /wp-sitemap.xml, BigCommerce /xmlsitemap.php, Next.js App Router /sitemap/0.xml); Deep mode tests 70+ CMS-specific and localized international paths.
• Recursive Sitemap Index Expansion: Automatically traverses nested <sitemapindex> hierarchies (Shopify, Yoast SEO, RankMath) down to depth 2, discovering all child partition files.

2. Smart Dual-Engine Architecture (auto / http / browser)

• HTTP Mode: Pure native Node.js fetch() requests (~1.2s per domain) with Chrome desktop User-Agent and gzip decompression. Zero browser memory overhead.
• Browser Mode: Playwright stealth Chromium (CloakBrowser) for anti-bot hardened targets.
• Auto Mode (Default): Attempts lightning-fast HTTP first; automatically escalates to stealth Playwright Chromium upon encountering Cloudflare Turnstile, DataDome, Akamai, or AWS WAF challenge screens.

3. Anti-False-Positive & Soft-404 SPA Shield

Modern Single Page Applications (Next.js, Vite, React, Vue) return HTTP 200 OK with index.html for non-existent sitemap URLs. The plugin strictly validates genuine XML root tags (<urlset>, <sitemapindex>, <rss>, <feed>) and enforces MIME/payload contracts to eliminate false positives.

4. Deep Edge-Case Engineering

• Subpath & Dirty URL Sanitization: Automatically cleans dirty input URLs (e.g., linktr.ee/profile, domain.com/blog/article) to inspect the canonical root domain.
• UTF-8 BOM Sanitization: Automatically strips invisible Byte Order Marks (\uFEFF) common in Windows IIS and legacy CMS sitemaps.
• CDATA & Entity Unescaping: Unwraps <loc><![CDATA[...]]></loc> and decodes XML entities.
• Native Gzip Decompression: Transparently streams and decompresses .xml.gz sitemaps using native Node.js zlib.
• Cross-Domain CDN Support: Seamlessly parses sitemaps hosted on external cloud CDNs (AWS CloudFront, Cloudflare, S3, Google Cloud Storage, Fastly).
• Canonical Redirect Tracking: Follows 301/308 HTTP redirects and captures the true destination URL (response.url).

📊 Comparison with Alternatives

⚙️ How to Configure Your Job

Example Input Configuration

{
  "urls": "shopify.com, stripe.com, techcrunch.com",
  "scrapeMode": "auto",
  "deepScan": false,
  "expandIndexes": true,
  "concurrency": 10,
  "limit": 50
}

📥 Supported URL & Input Formats

• Bare Domains: shopify.com, stripe.com (automatically prepends https://).
• Standard HTTP/HTTPS URLs: https://www.apple.com, http://example.com/.
• Subpaths & Deep URLs: https://linktr.ee/brand, https://dhigrowth.com/about-us (automatically normalized to the canonical root origin domain).
• Delimited Domain Batches: Multiple domains separated by commas (,), semicolons (;), tabs, or line breaks (\n).

📋 Output Data Fields

Every discovered sitemap produces a clean, structured diagnostic record:

💡 Example Output

[
  {
    "inputUrl": "shopify.com",
    "domain": "shopify.com",
    "sitemapUrl": "https://shopify.com/sitemap.xml",
    "sitemapFound": true,
    "sitemapType": "index",
    "urlsCount": 14,
    "lastModified": "2026-10-05T12:00:00Z",
    "discoverySource": "robots.txt",
    "category": "index",
    "isGzipped": false,
    "fileSizeBytes": 1480,
    "scrapeMode": "http",
    "status": "found",
    "error": ""
  },
  {
    "inputUrl": "shopify.com",
    "domain": "shopify.com",
    "sitemapUrl": "https://shopify.com/sitemap_products_1.xml",
    "sitemapFound": true,
    "sitemapType": "urlset",
    "urlsCount": 4850,
    "lastModified": "2026-10-04T18:22:10Z",
    "discoverySource": "index_child",
    "category": "products",
    "isGzipped": false,
    "fileSizeBytes": 492010,
    "scrapeMode": "http",
    "status": "found",
    "error": ""
  },
  {
    "inputUrl": "small-local-shop.test",
    "domain": "small-local-shop.test",
    "sitemapUrl": "",
    "sitemapFound": false,
    "sitemapType": "none",
    "urlsCount": 0,
    "lastModified": "",
    "discoverySource": "none",
    "category": "none",
    "isGzipped": false,
    "fileSizeBytes": 0,
    "scrapeMode": "http",
    "status": "not_found",
    "error": ""
  }
]

💼 Common Use Cases

• Technical SEO Auditing: Rapidly audit sitemap existence, index structure, and lastmod freshness across client websites without desktop crawler bloat.
• E-Commerce Catalog Sizing: Inspect multi-partition product sitemaps on Shopify, BigCommerce, and Magento to determine competitor catalog size and new product drops.
• URL Harvesting for Data Pipelines: Extract clean, canonical URL inventories from websites to feed downstream detail scrapers and LLM training sets.
• Migration & Redesign Verification: Validate that all CMS post types, taxonomy categories, and localized pages are properly published and reachable after site migrations.

💡 Best Practices for Large Enterprise Audits

1. Enterprise E-Commerce Audits (Shopify, BigCommerce, Magento):

   • Keep expandIndexes: true to unpack multi-partition catalogs (sitemap_products_1.xml, sitemap_products_2.xml, etc.).
   • Use deepScan: true if you suspect the platform uses non-standard media or custom CDN endpoints.

2. High-Volume Domain Sweeps (1,000+ domains):

   • Set scrapeMode: "http" to run at maximum throughput (~1.2s per site) with zero browser memory overhead.
   • Use concurrency: 15 or 20 to complete large domain batches in minutes.

3. Cloudflare & Anti-Bot Protected Domains:

   • Keep default scrapeMode: "auto". The detector probes via HTTP first and seamlessly switches to stealth Playwright Chromium only if a challenge header or CAPTCHA screen is detected.

4. Integration with Downstream Scraping:

   • Sitemaps provide an authoritative, clean inventory of live canonical URLs. Feed the sitemapUrl records into downstream detail scrapers for zero-waste data extraction.

❓ Frequently Asked Questions (FAQ)

What is the difference between a sitemap index and a regular sitemap?

A regular sitemap (<urlset>) contains a direct list of web page URLs. A sitemap index (<sitemapindex>) is a parent master directory that references multiple child sitemap files. Sitemap Detector automatically detects both types and unpacks nested sitemaps recursively.

Can Sitemap Detector extract gzipped .xml.gz sitemaps?

Yes. Many high-traffic websites and e-commerce stores compress sitemaps using gzip to reduce file size. The detector transparently streams and decompresses .xml.gz archives in memory using native Node.js zlib without requiring external utilities.

How does the detector prevent false positives on modern SPAs?

Modern Single Page Applications (built with Next.js, Vite, React, or Vue) often return an HTTP 200 OK status with an HTML error shell for any URL. Sitemap Detector uses strict structural XML validation contracts, confirming root tag validity (<urlset>, <sitemapindex>, <rss>, <feed>) and rejecting HTML doctype payloads.

Does it support non-standard sitemap locations?

Yes. While standard sitemaps live at /sitemap.xml, the plugin uses a 5-stage discovery engine that inspects robots.txt, HTML <head> tags, HTTP Link headers, and a curated probing dictionary of 70+ CMS paths (including Next.js /sitemap/0.xml, BigCommerce /xmlsitemap.php, and localized international folders).