# V³ Aware (v3.news), robots.txt # # Brand: v3.news is canonical; velox.news / versutus.news / vigilans.news # are branded aliases serving the same content. # on every page points back to v3.news so search engines consolidate # authority there. # # Crawler policy: maximum-reach allowlist on public surfaces, operator # routes disallowed for everyone. See MARKETING.md §1.6 for rationale. # Abusive scrapers (ignore UA, ignore robots) are handled at the edge # (Cloudflare in front of Render, and API rate limits), out of scope for this file. # # Rev 2026-10-26 (SEO/AEO review): the social and OG preview images under # /api/og/ and /api/social/ are allowed (link-preview crawlers that honour # robots.txt were refused them by the blanket /api/ rule), and /entities/ # is open (entity pages have a public part since 2026-10-15). /_next/ is no # longer disallowed: it holds the CSS and JavaScript Google renders pages # with (Search Console listed 83 blocked stylesheets; Google advises never # blocking them). # ── Default rule: all unlisted bots get the same allowlist ───────────── User-agent: * Allow: / Allow: /events Allow: /events/ Allow: /briefs Allow: /briefs/ Allow: /categories Allow: /sources Allow: /pricing Disallow: /sign-in Disallow: /sign-up Disallow: /admin Disallow: /admin/ Disallow: /monitor Disallow: /changelog Disallow: /risk Disallow: /map Disallow: /map/ Disallow: /search Disallow: /signals Disallow: /me Disallow: /me/ Allow: /api/og/ Allow: /api/social/ Disallow: /api/ Disallow: /lab Disallow: /lab/ Crawl-delay: 2 # ── Named AI / search crawlers, explicit allowlist for clarity ─────── # Each block re-states the same public surface so the bot can't # misinterpret an unset rule. Order: AI assistants, then AI search, # then traditional search, then archives. User-agent: GPTBot Allow: / Allow: /events Allow: /events/ Allow: /briefs Allow: /briefs/ Allow: /categories Allow: /sources Disallow: /admin Disallow: /admin/ Disallow: /monitor Disallow: /changelog Disallow: /risk Disallow: /map Disallow: /map/ Disallow: /search Disallow: /signals Disallow: /me Disallow: /me/ Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Claude-User Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Claude-SearchBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Claude-Web Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: anthropic-ai Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Perplexity-User Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Googlebot-News Allow: / Allow: /events Allow: /events/ Allow: /briefs Allow: /briefs/ Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Bingbot Allow: / Allow: /events Allow: /events/ Allow: /briefs Allow: /briefs/ Allow: /categories Allow: /sources Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: DuckDuckBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: CCBot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Bytespider Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ User-agent: Amazonbot Allow: / Disallow: /admin Disallow: /monitor Disallow: /risk Disallow: /map Disallow: /search Disallow: /signals Disallow: /me Allow: /api/og/ Allow: /api/social/ Disallow: /api/ Sitemap: https://v3.news/sitemap.xml Sitemap: https://v3.news/news-sitemap.xml