Robots.txt & AI Crawler Access Auditor

robots.txt Tester, RFC 9309 Parser & AI Crawler Checker

An enterprise-grade, high-performance IETF RFC 9309 robots.txt validator, 2026 AI crawler block auditor, sensitive endpoint vulnerability scanner, and 3-way crawlability cross-check engine for BrowserMesh.

Audits website crawler access policies across 30+ major bots, detects sensitive internal path disclosures (.env, /admin, /backup, /api/internal), performs active HTTP 200 exposure verification, inspects homepage X-Robots-Tag and HTML <meta name="robots"> tags for conflicting signals, and evaluates 2026 AI readiness standards (llms.txt, EU AI Act TDMrep, and Content-Signal).

The definitive modern alternative to Screaming Frog SEO Spider (£209/yr), Merkle Technical SEO Robots Tester, Google Search Console Robots Testing Tool, and brittle DIY Python scripts.

🎯 Target Audience & Industry Solutions

• Technical SEO Specialists & Webmasters: Verify that search engines (Googlebot, Bingbot, Applebot) can crawl critical content without accidental Disallow: / locks, soft-404 crawl traps, or conflicting X-Robots-Tag: noindex headers.
• AI Strategists & Content Publishers: Audit whether your content is protected from foundation LLM training scrapers (GPTBot, ClaudeBot, Google-Extended, DeepSeekBot, Bytespider) while keeping high-intent AI search and citation fetchers (OAI-SearchBot, PerplexityBot) open for referral traffic.
• Cybersecurity & DevSecOps Teams: Detect sensitive internal paths, database backups, Git repositories, and admin dashboards accidentally advertised to attackers via public Disallow: directives ("security by obscurity").
• Legal & Compliance Officers (EU AI Act Article 53): Verify machine-readable Text and Data Mining reservations (TDM-Reservation: 1 and /.well-known/tdmrep.json) to enforce copyright reservations against GPAI model providers.
• Digital Agencies & Enterprise Auditing: Perform batch audits across thousands of client domains concurrently with structured JSON output and objective 0–100 health scoring.

🚀 Key Features

1. Strict RFC 9309 IETF Compliance

Built directly against the September 2022 IETF Robots Exclusion Protocol standard:

• Longest Octet Match Precedence: Automatically calculates rule octet length to determine winning directive.
• RFC 9309 §2.2.2 Tie-Breaker: When Allow and Disallow rules match the exact same character length, Allow takes precedence.
• Universal Wildcards & Anchors: Accurately processes * (zero or more characters) and $ (end-of-URI anchor).
• Universal Fallback: Follows specific crawler blocks when present, falling back to User-agent: * only when required.
• 500 KiB Cutoff Safety: Enforces the 512,000-byte cutoff limit enforced by major crawlers.
• UTF-8 BOM Sanitization: Automatically strips Byte Order Marks (\uFEFF).

2. 2026 3-Tier AI Crawler Governance (30+ Bots)

Classifies and evaluates crawlers across modern operational tiers:

• AI Training Scrapers (Pre-Training): GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, DeepSeekBot, GrokBot, CCBot, Meta-ExternalAgent, cohere-training-data-crawler, Amazonbot, Diffbot, MistralBot.
• AI Search & Real-Time Citations: OAI-SearchBot (ChatGPT Search), ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalFetcher, Amzn-SearchBot, Bingbot.
• Traditional Search Engines: Googlebot, Googlebot-Image, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider, Slurp, Sogou.
• SEO & Auditing Bots: AhrefsBot, SemrushBot, DotBot, MJ12bot.
• Archivers: ia_archiver (Internet Archive).

3. Sensitive Endpoint Vulnerability & Reachability Scanner

Scans Disallow: directives for leaked sensitive assets:

• Critical Risk: /.env, /.git, /.svn, /backup, /database.sql, /dump, /wp-config.php, /server-status.
• Warning Risk: /admin, /wp-admin, /dashboard, /cpanel, /api/internal, /api/private, /swagger, /actuator.
• Active HTTP 200 Reachability Probe: Performs lightweight HEAD/GET checks on top sensitive paths to report whether they are actually exposed (200 OK) vs protected (401/403) vs not found (404).

4. 3-Way Crawlability Verification

Cross-references crawler permissions across all three layers of crawl truth:

1. robots.txt protocol rules.
2. Root page HTTP response headers (X-Robots-Tag: noindex, user-agent scoped headers).
3. Root page HTML <meta name="robots"> and <meta name="googlebot"> tags. Catches classic SEO contradiction traps where a page is blocked in robots.txt while having a noindex tag that crawlers can never read.

5. 2026 AI Ecosystem & Compliance Signals

• Probes /llms.txt and /llms-full.txt markdown guidance files.
• Inspects EU AI Act Article 53 TDM-Reservation headers and /.well-known/tdmrep.json.
• Detects emerging Content-Signal: search=yes, ai-train=no directives.

6. Calibrated 0–100 Health Scoring Engine

• Features weighted deductions with safety caps (critical: max 60 pts, warning: max 25 pts).
• Severity-based ceiling caps ensure that sites with critical errors (such as total search engine blocks or exposed .env files) cannot receive an "Excellent" rating.
• Distinguishes intentional AI training blocks (0 pt deduction) from accidental search engine de-indexing (20 pt deduction).

📊 Comparison with Existing Alternatives

⚙️ Configuration & Inputs

Example Input Configuration

{
  "urls": "https://openai.com\nhttps://github.com\nstripe.com",
  "scrapeMode": "auto",
  "testPaths": "/, /blog, /admin, /api",
  "checkSensitivePaths": true,
  "checkRootMetaRobots": true,
  "checkAiEcosystem": true,
  "concurrency": 8,
  "timeout": 30,
  "limit": 50
}

📥 Supported URL & Input Formats

• Bare Domains: openai.com, stripe.com (automatically resolves to https://domain.com/robots.txt).
• Standard HTTP/HTTPS URLs: https://github.com, http://example.com/.
• Deep URLs & Subpaths: https://example.com/blog/article-1 (automatically targets the root domain's robots.txt and tests /blog/article-1 path permissibility).
• Direct robots.txt URLs: https://example.com/robots.txt.
• Multiline & Comma-Separated Batches: Paste hundreds of client domains directly from CSV or spreadsheets.

📤 Output Schema Reference (20 Contract Fields)

🛡️ Common robots.txt SEO Issues Detected

1. Accidental Full-Site Disallow (Disallow: /): Search engines like Googlebot and Bingbot are locked out from crawling, causing catastrophic organic traffic loss.
2. Conflicting robots.txt and Meta noindex Directives: If a page is disallowed in robots.txt, search engines can never crawl the page to see the <meta name="robots" content="noindex"> tag. The URL can still be indexed in Google search as a bare URL snippet.
3. HTML Soft-404 Fallback Screens: Web servers returning HTTP 200 OK with HTML content for missing /robots.txt files confuse search engines, leading to erratic indexing behavior.
4. Internal Admin & Secret Path Leakage: Exposing sensitive paths (/.env, /wp-admin, /backup, /database.sql) in public Disallow: directives alerts malicious scanners to internal routes.
5. Universal AI Scraper Over-Blocking: Blocking AI search engines (OAI-SearchBot, PerplexityBot) while intending to only block AI model pre-training scrapers (GPTBot, ClaudeBot), cutting off AI search referral traffic.

💡 Practical AI Governance Recipes

Scenario A: Allow AI Search Citations, Block AI Model Training

To allow ChatGPT Search and Perplexity to index your site for citations while preventing OpenAI and Anthropic from using your articles for LLM pre-training:

# Block foundation model pre-training scrapers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Keep real-time AI citation & search bots open
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Keep organic search engines open
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

❓ Frequently Asked Questions (FAQ)

What is IETF RFC 9309 and why does it matter for SEO?

Published in September 2022, RFC 9309 is the official IETF standard for the Robots Exclusion Protocol. It establishes standard rule precedence (longest octet match wins, and Allow wins ties) and limits files to 500 KiB. Following RFC 9309 guarantees your robots.txt file is parsed identically across Google, Bing, Apple, and AI search engines.

Does blocking GPTBot hurt my Google Search rankings?

No. GPTBot is operated solely by OpenAI for pre-training models like GPT-4o. Google uses Googlebot for search indexing and Google-Extended for Gemini AI training. Blocking GPTBot has zero impact on Google organic search.

What is the difference between GPTBot and OAI-SearchBot?

• GPTBot: Scrapes web data for OpenAI foundation model training.
• OAI-SearchBot: Crawls pages specifically for ChatGPT Search (SearchGPT) to deliver real-time answers with source attribution links. Blocking OAI-SearchBot will prevent your site from appearing as a cited source in ChatGPT Search.

How do I allow Perplexity citations without giving away AI training data?

Allow PerplexityBot while disallowing bulk crawlers like CCBot (Common Crawl), Bytespider, and Diffbot. Perplexity uses PerplexityBot to index pages and provide cited answers to users in real time.

Why does the checker report a critical issue for HTML soft-404s?

When a web server returns HTTP 200 OK for /robots.txt but actually serves an HTML 404 page or Single Page Application shell, crawlers attempt to interpret the HTML tags as robots.txt directives. This can lead to unpredictable crawling behavior and search engine de-indexing.

What is the EU AI Act TDM-Reservation?

Under Article 53 of the EU AI Act (enforceable August 2026), General Purpose AI (GPAI) providers must respect copyright reservations made by rightsholders using machine-readable standards like W3C TDMrep (TDM-Reservation: 1 header or /.well-known/tdmrep.json).

Can Disallow rules expose sensitive URLs to attackers?

Yes. Directives like Disallow: /admin-secret/ or Disallow: /backup.sql tell everyone where your private assets are located. Security through obscurity is a high risk. Sensitive endpoints must be protected with authentication and firewall rules, not advertised in robots.txt.