Sitemap Extractor

Sitemap Extractor & URL Inventory Audit Engine

An enterprise-grade, memory-safe XML sitemap parser, nested sitemap index crawler, and technical SEO URL extraction engine for BrowserMesh.

Point it at any website domain or direct sitemap link. Automatically inspects robots.txt (IETF RFC 9309), resolves sitemap directives, probes high-probability CMS paths, recursively traverses multi-level sitemap indexes, transparently decompresses .xml.gz payloads, and streams clean itemized page URLs with publisher metadata (lastmod, changefreq, priority, image sitemaps) alongside an aggregated site audit summary.

The premier high-throughput, memory-safe alternative to Apify Sitemap Extractor, Screaming Frog XML Sitemap Crawls, and memory-heavy DOM XML parsers.

🎯 Target Audience & Industry Solutions

• Technical SEO Directors & Enterprise Crawlers: Extract complete canonical URL inventories across 50,000+ page domains to benchmark indexation coverage, calculate content freshness velocity, audit publisher <lastmod> reliability, and seed deep architectural crawls.
• Data Engineering & Web Scraping Architects: Generate ready-made, pre-filtered URL frontiers for downstream web scraping pipelines without performing expensive in-browser link discovery or recursive DOM rendering.
• E-Commerce & CMS Migration Teams: Audit and catalogue legacy URL paths before and after major CMS platform migrations (Shopify, WordPress, Webflow, Magento, BigCommerce) to guarantee 100% 301-redirect mapping coverage and prevent 404 traffic cliffs.
• RAG & Generative AI Pipeline Engineers: Feed clean, canonical document endpoints directly into Large Language Model (LLM) embeddings, vector databases (Pinecone, Weaviate, Qdrant), and document ingestion loaders.
• Competitive Intelligence & Publishing Analysts: Track publication velocity, new product drops, and editorial update cadences of competitors by monitoring changes in <lastmod> timestamps over recurring scheduled runs.

⚔️ Architectural Comparison

🚀 Key Features

1. Memory-Safe Streaming URL Extraction Architecture

Enterprise eCommerce and media websites frequently publish XML sitemaps containing up to 50,000 URLs per file (up to 50MB uncompressed). Traditional XML parsers parse the entire document into massive Document Object Model (DOM) AST trees, creating severe memory spikes and risking V8 Out-Of-Memory (OOM) crashes.

• Chunked Tag Scanner: Scans <url>...</url> blocks sequentially via high-performance RegExp streaming.
• Sub-25MB Footprint: Keeps memory usage virtually flat regardless of sitemap size.
• Live Stream Yields: Discovered URLs are yielded immediately (type: 'url') so downstream queues process records without waiting for massive jobs to finish.

2. Recursive Sitemap Index Traversal & Loop Shield

• Deep Hierarchical Crawling: Automatically identifies <sitemapindex> roots and enqueues all declared child <sitemap><loc> paths up to 4 levels deep.
• Circular Reference Protection: Maintains an in-memory visitedSitemaps set to neutralize recursive indexing loops (e.g. Sitemap A referencing Sitemap B which references Sitemap A).
• Run-Level Deduplication: Ensures duplicate URL declarations across multiple category or post sitemaps are emitted exactly once.

3. Dual-Engine Intelligent WAF Escalation

• Native HTTP Engine: Dispatches ultra-fast native HTTP requests with optimized headers, brotli/gzip handling, and sub-300ms latency per sitemap.
• Playwright Stealth Browser Escalation: In auto mode, when an endpoint returns 403 Forbidden, 503 Service Unavailable, Cloudflare Turnstile, or DataDome challenge signatures, the plugin automatically escalates the request to a stealth Chromium page to bypass the bot wall.

4. RFC 9309 Auto-Discovery & Probing Engine

• Direct Sitemap Detection: Automatically bypasses discovery if a direct .xml, .xml.gz, .txt, .atom, or /sitemap link is provided.
• Robots.txt Directive Parsing: Conforms strictly to IETF RFC 9309, stripping comments, resolving relative sitemap links, and extracting case-insensitive Sitemap: directives.
• Universal CMS Probing: Probes top universal CMS sitemap conventions (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap_products_1.xml) if no robots.txt directives exist.

5. Publisher Metadata & Google Extensions

• Change Metadata: Extracts publisher <lastmod>, <changefreq>, and <priority> values when present.
• Date Filtering (modifiedSince): Excludes URLs with parseable dates older than a specified cutoff date, while retaining unknown/undated entries per RFC guidelines.
• Google Image Sitemaps: Extracts declared <image:loc> image assets when includeImages: true.

6. Dual-Record Output Architecture

• Itemized Page Records (type: 'url'): Granular, individual records streamed for each discovered URL.
• Final Site Audit Summary (type: 'summary'): An executive-level aggregated record emitted upon completing each target website, reporting total unique URLs, sitemaps processed, breakdown by sitemap file, total images, and crawl execution duration.

📋 Input Configuration Parameters

📊 Structured Output Fields Reference (Domain Record)

Nested sitemaps Item Format:

{
  "sitemapSource": "https://example.com/sitemap.xml",
  "sitemapType": "urlset",
  "urlsCount": 54,
  "urls": [
    {
      "url": "https://example.com/page-1",
      "lastModified": "2026-10-11T15:16:42.991Z",
      "changeFreq": "daily",
      "priority": "0.8",
      "imagesCount": 0
    }
  ]
}

💡 Best Practices for Enterprise Crawls

1. Direct Sitemap Links vs Root Domains:
   • For fastest execution, provide direct sitemap links (e.g. https://example.com/sitemap_products_1.xml) to bypass robots.txt discovery and path probing.
2. Cap Sizing with maxResults:
   • For quick checks or reconnaissance runs, keep maxResults at 100–1000. Set to -1 only when conducting complete domain inventory audits.
3. Filtering by Modification Date:
   • Use modifiedSince (e.g. 2025-01-01) to extract only newly added or updated pages, drastically accelerating recurring ingestion pipelines.
4. Cloudflare & Anti-Bot Protected Sites:
   • Leave scrapeMode on 'auto'. It consumes zero browser resources on normal sites, but seamlessly escalates to stealth Chromium when WAF challenges appear.

❓ Frequently Asked Questions (FAQ)

Q: How does this plugin handle compressed .xml.gz sitemaps?

Transparently. The plugin inspects response headers and gzip magic bytes (0x1f, 0x8b), decompressing payloads in-memory using native Node.js zlib before passing strings to the streaming parser.

Q: What happens if a website does not publish any sitemap?

The plugin does not crash or silently omit the domain. It yields one diagnostic record with status: 'error' and error: 'No usable sitemap found...' so input records align cleanly in downstream CSV/Excel exports. If includeSummary is enabled, a diagnostic summary record is also yielded.

Q: Does date filtering discard pages with missing <lastmod> tags?

No. Following RFC standards and search engine best practices, entries with missing or unparseable <lastmod> tags are retained because their modification date is unknown. Only entries with an explicitly parseable date older than the cutoff are filtered.

Q: Can I run this plugin without installing Playwright or extra dependencies?

Yes. The plugin is configured with "needOwnDependencies": false. It uses native Node.js fetch and zlib, keeping deployment instant and dependency-free. Stealth browser fallback is only activated if Playwright page context is supplied by the runtime.