Website Meta Tag Extractor

Website Meta Tag Extractor & SEO Health Auditor

An enterprise-grade, high-throughput Website Meta Tag Extractor, Open Graph Inspector, Schema.org Structured Data Parser, and Calibrated SEO Health Auditor for BrowserMesh.

Discovers, extracts, normalizes, validates, and audits over 150+ metadata descriptors across 10 modern web standard families. Concurrently executes an objective 0–100 diagnostic health scoring engine, audits modern 2026 AI bot directives (GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended, PerplexityBot), discovers llms.txt v2 machine-readable context pointers, and verifies ActivityPub Fediverse attribution (fediverse:creator, rel="me").

The definitive modern 2026 alternative to Screaming Frog SEO Spider (£209/yr), Metatags.io, Ahrefs Site Audit, and single-threaded scrapers.

🎯 Target Audience & Real-World Use Cases

• Technical SEO Directors & Web Quality Auditors: Audit titles, meta descriptions, canonical URLs, robots indexing permissions, and hreflang language tags across entire domains via direct URLs or automated XML sitemap traversal.
• Generative Engine Optimization (GEO) & AI Search Marketers: Ensure your website is optimized for modern AI search engines (ChatGPT Search, Perplexity AI, Google Gemini, Claude) by auditing live citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) vs. model training opt-outs (GPTBot, ClaudeBot, Google-Extended), alongside llms.txt v2 machine-readable context discovery.
• Social Media Managers & Growth Engineers: Verify Open Graph and Twitter/X Card tags to guarantee rich link previews unfurl reliably across Facebook, X/Twitter, LinkedIn, Slack, WhatsApp, and Discord with optimal 1200×630px dimensions and sub-300KB mobile payloads.
• Decentralized Social & Fediverse Creators: Verify ActivityPub fediverse:creator attribution and rel="me" profile verification links across Threads, Mastodon, and Ghost to capture reader follow cards and author bylines.
• Frontend & Jamstack Engineers: Audit Next.js, Remix, Vite, Nuxt, and SvelteKit single-page applications with dual-engine execution (Fast HTTP + automatic stealth Chromium escalation) to eliminate the "SPA hydration blindspot."
• Academic Publishers & Research Portals: Extract Google Scholar, Highwire Press (citation_*), PRISM, and Dublin Core metadata to ensure scholarly articles and whitepapers index seamlessly.

🚀 Key Features & Capabilities

1. 10 Comprehensive Metadata Standards Families

• HTML5 Core Document Metadata: <title>, <meta name="description">, keywords, robots indexing directives (index, follow, noimageindex, nosnippet, max-snippet, max-image-preview:large, indexifembedded), viewport responsive data, charset, CMS generator, light/dark theme colors (media="(prefers-color-scheme: ...)"), color-scheme, and format detection.
• 2026 AI Search & LLM Directives: Granular extraction of 24 AI bot directives across OpenAI, Anthropic, Google, Perplexity, Apple, Meta, ByteDance, DeepSeek, and xAI.
• Open Graph Protocol & Structured Media: Complete extraction of core OG tags, vertical schemas (article, product, book, profile, music), and sequential image/video/audio property pairing with Zakas state-machine resolution.
• Twitter / X Cards: Full card validation (summary, summary_large_image, app, player), checking both name="twitter:*" and property="twitter:*" attributes, media resolution, and app deep links.
• Schema.org JSON-LD & Microdata: Unpacks @graph arrays and multi-root schema objects (Article, Organization, LocalBusiness, Person, Product) while flagging 2026 retired SERP rich result types (FAQPage, HowTo).
• Mobile Deep Linking & Fediverse Attribution: Apple iOS Smart App Banners, Facebook App Links (al:*), and ActivityPub decentralized attribution (fediverse:creator, rel="me").
• Academic & Publisher Citations: Highwire Press (citation_*), Google Scholar, PRISM, and Dublin Core (dc:*, dcterms.*).
• Discovery, Feeds & Favicons: RSS, Atom, JSON Feeds, llms.txt v2 discovery (rel="describedby" and type="text/markdown"), Web App Manifests, and high-resolution favicon ranking (SVG prioritized with 100,000 score).
• Verification Tokens: Google, Bing/Microsoft, Yandex, Facebook, Pinterest, and Ahrefs site verification tokens.
• Complete Raw Meta Dictionary: Verbatim dictionary of every <meta> tag in the document.

2. Dual-Engine Architecture (auto / http / browser)

• Fast HTTP Engine (http): Native fetch with modern Chrome 131 Desktop headers and Gzip decompression (~150–300ms per page, zero browser memory overhead).
• Stealth Playwright Chromium Engine (browser): Fully headless Chromium anti-bot bypass for client-hydrated SPAs and anti-bot challenge pages.
• Auto Engine (auto, Default): Executes fast HTTP first; automatically escalates to stealth Chromium if an empty title, generic SPA shell ("Loading...", "Vite App"), or Cloudflare challenge is encountered.

3. XML Sitemap Crawler & Smart Slot Rollup

• Crawls .xml, .xml.gz, and <sitemapindex> hierarchies with automatic Gzip decompression.
• Probes /sitemap_index.xml, /wp-sitemap.xml, or crawls homepage links if the primary sitemap returns 404.
• Smart Slot Reservation: When limit = N, audits $N - 1$ pages and yields the $N$-th record as the consolidated [Site Audit Summary] rollup.

🤖 2026 Generative Engine Optimization (GEO) & AI Bot Catalog

The extractor audits robots directives for all 24 modern AI bot user-agents, classifying them into foundation model training scrapers vs. real-time search citation bots:

🥊 Competitive Comparison

🏆 Calibrated 0–100 Health Scoring Engine & Rubric

Every page begins with a baseline score of 100 points. Deductions are applied for missing tags, sub-optimal character lengths, and social unfurling bugs. Progressive enhancement bonuses (+5 pts each) are awarded when the pre-bonus score is $\ge 85$.

Letter Grade Scale

Diagnostic Deductions & Safety Caps

Safety Caps & Ceiling Rules:

• Length Deductions Cap: Capped at -15 pts maximum to avoid penalizing lengthy copy excessively.
• Social Deductions Cap: Capped at -30 pts maximum so social omissions do not mask technical crawlability.
• Critical Ceiling Cap: If <title> or <link rel="canonical"> is missing, the overall score is hard-capped at $\le 75$ (Grade C) regardless of other tags.

Progressive Enhancement Bonuses (+5 pts each, pre-bonus score $\ge 85$)

• has_json_ld (+5 pts): Document implements valid Schema.org JSON-LD structured data.
• has_hreflang (+5 pts): Implements international multilingual annotations (rel="alternate" hreflang="...").
• has_favicon_suite (+5 pts): Modern icon suite containing SVG icons or Apple Touch Icons (180×180px).
• has_web_manifest (+5 pts): Links to a Web App Manifest (<link rel="manifest">).
• has_llms_txt (+5 pts): Implements modern llms.txt v2 discovery or Markdown alternate pointers.
• has_fediverse_creator (+5 pts): Implements ActivityPub fediverse:creator attribution for Mastodon/Threads.

📋 Strict 25-Field Output Contract

Every yielded record strictly implements 25 structured fields:

📄 Real-World Output Sample JSON

1. Individual Webpage Audit Record

{
  "url": "https://www.browsermesh.in",
  "domain": "browsermesh.in",
  "statusCode": 200,
  "status": "pristine",
  "healthScore": 100,
  "grade": "A",
  "title": "BrowserMesh | Enterprise Browser Automation & RPA Platform",
  "description": "Self-hosted browser automation and RPA platform for Windows, Linux & Android. Run distributed residential and 4G mobile browser fleets with stealth fingerprinting.",
  "canonicalUrl": "https://www.browsermesh.in/",
  "language": "en",
  "basic": {
    "title": "BrowserMesh | Enterprise Browser Automation & RPA Platform",
    "description": "Self-hosted browser automation and RPA platform for Windows, Linux & Android. Run distributed residential and 4G mobile browser fleets with stealth fingerprinting.",
    "keywords": ["browser automation", "rpa", "playwright", "residential proxies"],
    "robots": {
      "index": true,
      "follow": true,
      "maxImagePreview": "large",
      "raw": "index, follow, max-image-preview:large"
    },
    "viewport": {
      "width": "device-width",
      "initialScale": "1.0",
      "raw": "width=device-width, initial-scale=1.0"
    },
    "charset": "utf-8"
  },
  "openGraph": {
    "title": "BrowserMesh | Enterprise Browser Automation & RPA Platform",
    "type": "website",
    "url": "https://www.browsermesh.in/",
    "images": [
      {
        "url": "https://www.browsermesh.in/opengraph-img.png",
        "width": 1200,
        "height": 630,
        "alt": "BrowserMesh Platform Banner"
      }
    ]
  },
  "twitterCard": {
    "card": "summary_large_image",
    "site": "@BrowserMesh",
    "image": "https://www.browsermesh.in/opengraph-img.png"
  },
  "jsonLd": [
    {
      "@context": "https://schema.org",
      "@type": "SoftwareApplication",
      "name": "BrowserMesh",
      "applicationCategory": "DeveloperApplication",
      "operatingSystem": "Windows, Linux, Android"
    }
  ],
  "audit": {
    "score": 100,
    "grade": "A",
    "deductions": [],
    "warnings": [],
    "bonuses": [
      {
        "code": "has_json_ld",
        "points": 5,
        "message": "Document includes valid Schema.org JSON-LD structured data."
      },
      {
        "code": "has_favicon_suite",
        "points": 5,
        "message": "Document includes modern high-resolution icon suite (SVG / Apple Touch Icon)."
      }
    ],
    "recommendations": []
  }
}

2. Consolidated Site Rollup Summary Record ([Site Audit Summary])

{
  "url": "browsermesh.in [Site Audit Summary]",
  "domain": "browsermesh.in",
  "statusCode": 200,
  "status": "completed",
  "healthScore": 100,
  "grade": "A",
  "title": "Site-Wide Metadata Audit Summary for browsermesh.in",
  "description": "Consolidated metadata health assessment across 54 audited pages.",
  "basic": {
    "totalPagesAudited": 54,
    "cleanPagesCount": 40,
    "warningPagesCount": 14,
    "issuePagesCount": 0,
    "averageHealthScore": 100,
    "siteGrade": "A",
    "missingTitleRate": 0,
    "missingDescriptionRate": 0,
    "missingCanonicalRate": 0,
    "missingOpenGraphRate": 0,
    "aiSearchOptInRate": 100,
    "llmsTxtDiscoveryRate": 0,
    "fediverseAttributionRate": 98.1
  },
  "audit": {
    "score": 100,
    "grade": "A",
    "topSiteIssues": [
      { "code": "title_too_long", "count": 8, "percentage": 14.8 },
      { "code": "description_too_long", "count": 7, "percentage": 13.0 },
      { "code": "duplicate_title_tags", "count": 1, "percentage": 1.9 }
    ],
    "recommendations": [
      "Shorten title to under 70 characters to prevent ellipsis truncation in Google SERPs.",
      "Shorten description to under 160 characters to prevent snippet cut-off.",
      "Consolidate multiple <title> tags into a single definitive document title."
    ]
  }
}

⚙️ Configuration Parameters Reference

💡 Best Practices for Technical SEO & Social Metadata

1. Always Define Explicit Open Graph Dimensions (1200×630px): Include both <meta property="og:image:width" content="1200"> and <meta property="og:image:height" content="630">. When a URL is shared on Facebook, LinkedIn, or iMessage for the first time, social crawlers download the image asynchronously. Without explicit dimensions, the crawler cannot determine the aspect ratio, resulting in a blank preview card for the initial sharer.

2. Never Use SVG Files for Social Share Previews: Major social platforms (Facebook, X/Twitter, LinkedIn, WhatsApp, Pinterest) reject .svg images in link preview unfurls due to security and rasterization restrictions. Always provide standard high-resolution .jpg, .png, or .webp assets.

3. Keep Titles Between 20 and 70 Characters: Google desktop and mobile search snippets truncate titles longer than ~65–70 characters (or ~600px display width), replacing trailing keywords with an ellipsis (...). Ensure your primary keyword and brand name fit comfortably within the 70-character threshold.

4. Target Meta Descriptions Between 50 and 160 Characters: Descriptions under 50 characters lack meaningful search intent context, prompting search engines to generate automated snippets from arbitrary page text. Conversely, descriptions over 160 characters get truncated on mobile devices.

5. Separate Search Citations (OAI-SearchBot) from Training Scrapers (GPTBot): In modern Generative Engine Optimization (GEO), blocking all AI bots harms visibility. Allow live search indexers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot to drive referral traffic and AI citations, while selectively disallowing pre-training scrapers (GPTBot, ClaudeBot, Google-Extended) to protect proprietary content.

6. Provide Self-Referencing Absolute Canonicals: Always specify a fully qualified absolute URL (e.g., https://example.com/page, not /page or relative paths) in <link rel="canonical"> to consolidate link equity across HTTP/HTTPS, www/non-www, trailing slashes, and UTM tracking parameters.

7. Consolidate Duplicate <title> Tags in the DOM: Ensure inline SVG icons or embedded components do not contain un-scoped <title> elements inside the HTML body or head, which can confuse crawler title parsers.

💬 Frequently Asked Questions (FAQ)

What makes BrowserMesh different from traditional meta tag extractors?

Traditional metadata scrapers only extract raw string attributes and frequently fail on client-rendered React/Next.js/Vue applications, missing dynamic tags completely. BrowserMesh uses a dual-engine architecture that executes high-speed HTTP fetch first and automatically escalates to stealth Chromium (CloakBrowser) when an SPA shell or anti-bot challenge is detected. Furthermore, it extracts modern 2026 standards (llms.txt v2, AI bot directives, and Fediverse creator attribution) and synthesizes an authoritative Golden Record with calibrated health scoring.

How does the 0–100 health scoring engine work?

The health scoring engine starts at 100 points and applies calibrated deductions for critical omissions (missing title, description, viewport, canonical, robots), length truncations, and social unfurling bugs (missing og:image:width/height, SVG social images). It enforces Category Safety Caps (Length deductions capped at 15 pts, Social capped at 30 pts) and a Severity Ceiling Cap (missing Title or Canonical caps the score at $\le 75$, Grade C). Documents with score $\ge 85$ receive progressive enhancement bonuses (+5 pts) for structured data, manifest, llms.txt, and Fediverse attribution.

Why does this plugin audit both OAI-SearchBot and GPTBot?

In 2026, AI search engines split their crawlers into distinct bot identities:

• OAI-SearchBot powers live ChatGPT search queries and web citations. Blocking it removes your site from ChatGPT search answers!
• GPTBot harvests training datasets to train future foundation models. Blocking it prevents model training while preserving search citations. BrowserMesh audits both so you can verify that your site allows citations while enforcing IP protection where desired.

Why are Open Graph image dimensions (og:image:width and og:image:height) essential?

When a link is shared on Facebook, LinkedIn, or iMessage for the very first time, the platform's crawler asynchronously downloads the image. If og:image:width (1200) and og:image:height (630) are missing, the social crawler cannot determine the aspect ratio and displays a blank preview box for the initial sharer. Specifying explicit dimensions allows instant full-bleed card rendering.

What is llms.txt v2 and how does it benefit Generative Engine Optimization (GEO)?

llms.txt is an emerging standard (similar to robots.txt) that provides large language models (LLMs) and AI agents with clean, machine-readable Markdown summaries of website capabilities. The plugin automatically detects llms.txt pointers via <link rel="describedby" href="/llms.txt"> and <link rel="alternate" type="text/markdown">, awarding a bonus for AI search readiness.