The short answer
Open Site SEO Check and AI Crawler Audit or connect https://sitecheck.openkrill.app/mcp to your AI assistant. Call check_ai_crawler_access to test 14 named AI crawlers against a site's robots.txt, check_page_tags to inspect its title, canonical URL and Open Graph tags, or ai_readiness_check to check llms.txt, sitemap.xml and the 5 page-level AI citation signals. For a sample multi-page report, see our AI search readiness audit page.
What our audit of theguardian.com found across 14 AI crawlers
We ran check_ai_crawler_access on https://www.theguardian.com/ on 2026-10-20. In 1,140 ms, the tool followed the HTTP 302 redirect to https://www.theguardian.com/uk (HTTP 200), read https://www.theguardian.com/robots.txt, noted the page's xRobotsTag: bingbot: noarchive and metaRobots: max-image-preview:large directives, and evaluated all 14 AI crawlers:
| Crawler | Operator | Purpose | Allowed in robots.txt | Matching rule |
|---|---|---|---|---|
GPTBot | OpenAI | Model training | Yes (true) | Falls under User-agent: * (no disallow on /) |
OAI-SearchBot | OpenAI | ChatGPT search | Yes (true) | Falls under User-agent: * |
ChatGPT-User | OpenAI | Page fetched for a ChatGPT user | Yes (true) | Falls under User-agent: * |
Perplexity-User | Perplexity | Page fetched for a Perplexity user | Yes (true) | Falls under User-agent: * |
Google-Extended | Gemini training and grounding | Yes (true) | Falls under User-agent: * | |
ClaudeBot | Anthropic | Model training | No (false) | Disallow: / |
Claude-SearchBot | Anthropic | Claude search | No (false) | Disallow: / |
Claude-User | Anthropic | Page fetched for a Claude user | No (false) | Disallow: / |
PerplexityBot | Perplexity | Perplexity search index | No (false) | Disallow: / |
Applebot-Extended | Apple | Apple model training | No (false) | Disallow: / |
Meta-ExternalAgent | Meta | Model training | No (false) | Disallow: / |
Amazonbot | Amazon | Amazon services and Alexa answers | No (false) | Disallow: / |
Bytespider | ByteDance | Model training | No (false) | Disallow: / |
CCBot | Common Crawl | Open web archive for model training | No (false) | Disallow: / |
Anthropic's 3 bots, PerplexityBot and CCBot are explicitly blocked with Disallow: /, while OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User are not named in the disallow group and inherit the open User-agent: * rules.
How Site SEO & AI Crawler Audit compared with rivals
We ran the same theguardian.com check against three competing tools on 2026-10-20:
| Tool | Result on theguardian.com | Wall time | Score |
|---|---|---|---|
| OpenKrill Site SEO & AI Crawler Audit | All 14 crawlers with operator, purpose, allowed status and matching rule; follows 302 to /uk; reports xRobotsTag and metaRobots | 1,140 ms | 10.0 / 10 |
| TechnicalSEO.com AI Bot Access Tool | Browser run checked 35 AI user agents with matching robots.txt rule and live HTTP status (403 on most blocked bots), but required a browser session and did not label each bot's operator or purpose | 4,485 ms | 9.1 / 10 |
SiteAudit MCP (vdalhambra/siteaudit-mcp) | check_robots_txt returned only the first 30 of 49 rules (all for *), cutting off the AI-bot disallow block completely so it could not tell whether GPTBot or ClaudeBot was blocked | 2,714 ms (2,146 ms startup + 568 ms call) | 5.2 / 10 |
| Apify AI Crawler Policy Audit | Not run: requires an Apify account and API token | not run | not run |
Checking Open Graph tags and AI citation signals on the same page
Two additional calls on https://www.theguardian.com/ show what else an agent can check in seconds:
- Page tags and head size (
check_page_tags). The tool read the 48-character title (Latest news, sport and opinion from the Guardian), the 112-character description, the canonical linkhttps://www.theguardian.com/and 6 icon links, and flaggedog:title,og:description,og:image,og:urlandtwitter:cardas missing in the first 150 KB of the HTML head (noting that tags placed after 150 KB of inline styles or scripts are not read). - AI readiness and citation checks (
ai_readiness_check). The domain servesrobots.txtandsitemap.xml(286 entries read fromsitemaps/news.xml), while/llms.txtand/agents.mdreturn HTTP 404. Across the 5 AI citation signals on the home page, 1 passed (Q&A format, with 12 question headings), 1 warned (54 lists and 0 tables, but no Schema.org JSON-LD), and 3 failed (no summary block near the top, no author byline on the front page, and noh1heading).
Connect it to your AI client
Add https://sitecheck.openkrill.app/mcp to your assistant, or follow the install steps for ChatGPT, Claude, Cursor and Pi. The 3 agent audit workflows are downloadable at site-check-jobs/SKILL.md.
Here is the JSON-RPC call from our crawler audit:
{"method":"tools/call","params":{"name":"check_ai_crawler_access",
"arguments":{"url":"https://www.theguardian.com/"}}}Mistakes to avoid when auditing AI crawlers
- Assuming one bot name per AI company. OpenAI, Anthropic and Perplexity use separate user agents for model training (
GPTBot,ClaudeBot), search indexing (OAI-SearchBot,Claude-SearchBot,PerplexityBot) and user-triggered page fetches (ChatGPT-User,Claude-User,Perplexity-User). - Confusing
robots.txtwith CDN firewall blocks.check_ai_crawler_accessreads the publishedrobots.txtrules and page directives; a CDN or WAF can still block a bot by IP or user agent even whenrobots.txtallows it. - Burying Open Graph tags past 150 KB of inline CSS or state scripts. Link-preview fetchers and
check_page_tagsstop reading oversized<head>blocks; keep meta and Open Graph tags near the top of<head>.
Frequently asked questions
Does my website's robots.txt block GPTBot, ClaudeBot or other AI crawlers?
Call check_ai_crawler_access on sitecheck.openkrill.app/mcp with your page URL. It tests 14 named AI crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended) and shows the exact robots.txt rule, X-Robots-Tag header and meta robots tag that apply to each one.
How can I check which AI bots are allowed to crawl a site?
Paste the URL into tools.openkrill.app/site-check or call check_ai_crawler_access and ai_readiness_check on https://sitecheck.openkrill.app/mcp. You get a per-crawler table with the operator, purpose (training, search index or user fetch) and whether robots.txt allows it.
Which free tool audits a page for AI search crawler access and SEO tags?
OpenKrill's Site SEO & AI Crawler Audit (tools.openkrill.app/tools/site-check) is free and needs no account or API key. It checks 14 AI crawlers in robots.txt, Open Graph and canonical tags, redirect chains, llms.txt and sitemap presence, and 5 AI citation signals.
Why does check_page_tags say my Open Graph tags are missing when I can see them in DevTools?
Either the tags are injected by client-side JavaScript (which social unfurlers and AI crawlers do not run) or they sit after the first 150 KB of the HTML head. Move your title, description, canonical and og:* tags to the top of the server-rendered head.
Sources
- Crawler access, page-tag and AI readiness runs: Site SEO & AI Crawler Audit (
https://sitecheck.openkrill.app/mcp), read on 2026-10-20 between 06:12 and 07:16 UTC. - The Guardian robots.txt file: theguardian.com/robots.txt, read on 2026-10-20.
- Competitors tested: TechnicalSEO.com AI Bot Access Testing Tool and vdalhambra/siteaudit-mcp, tested on 2026-10-20.