[Agent Tools](https://tools.openkrill.app/) / [Blog](https://tools.openkrill.app/blog)

# How to check which AI crawlers your robots.txt blocks and audit a page's share tags and AI citation signals

Many websites allow Googlebot while blocking AI training bots, or think they blocked every AI crawler when they left `GPTBot` and `OAI-SearchBot` wide open under `User-agent: *`. Worse, some SEO audit MCP servers truncate `robots.txt` after 30 lines and miss the AI crawler block at the bottom of the file. This guide shows what our live audit of `theguardian.com` found across 14 AI crawlers, Open Graph tags and AI citation signals, and compares Site SEO & AI Crawler Audit against two rival tools.

By OpenKrill. Published 2026-10-20. Every crawler rule, header and page-tag finding below comes from live calls to [Site SEO & AI Crawler Audit](https://tools.openkrill.app/tools/site-check) (`sitecheck.openkrill.app`) and direct reads of `theguardian.com` between 06:12 and 07:16 UTC on 2026-10-20.

## The short answer

Open [Site SEO Check and AI Crawler Audit](https://tools.openkrill.app/site-check) or connect `https://sitecheck.openkrill.app/mcp` to your AI assistant. Call `check_ai_crawler_access` to test 14 named AI crawlers against a site's `robots.txt`, `check_page_tags` to inspect its title, canonical URL and Open Graph tags, or `ai_readiness_check` to check `llms.txt`, `sitemap.xml` and the 5 page-level AI citation signals. For a sample multi-page report, see our [AI search readiness audit page](https://tools.openkrill.app/aiready).

## What our audit of theguardian.com found across 14 AI crawlers

We ran `check_ai_crawler_access` on `https://www.theguardian.com/` on 2026-10-20. In 1,140 ms, the tool followed the HTTP 302 redirect to `https://www.theguardian.com/uk` (HTTP 200), read `https://www.theguardian.com/robots.txt`, noted the page's `xRobotsTag: bingbot: noarchive` and `metaRobots: max-image-preview:large` directives, and evaluated all 14 AI crawlers:

| Crawler | Operator | Purpose | Allowed in robots.txt | Matching rule |
| --- | --- | --- | --- | --- |
| `GPTBot` | OpenAI | Model training | **Yes** (`true`) | Falls under `User-agent: *` (no disallow on `/`) |
| `OAI-SearchBot` | OpenAI | ChatGPT search | **Yes** (`true`) | Falls under `User-agent: *` |
| `ChatGPT-User` | OpenAI | Page fetched for a ChatGPT user | **Yes** (`true`) | Falls under `User-agent: *` |
| `Perplexity-User` | Perplexity | Page fetched for a Perplexity user | **Yes** (`true`) | Falls under `User-agent: *` |
| `Google-Extended` | Google | Gemini training and grounding | **Yes** (`true`) | Falls under `User-agent: *` |
| `ClaudeBot` | Anthropic | Model training | **No** (`false`) | `Disallow: /` |
| `Claude-SearchBot` | Anthropic | Claude search | **No** (`false`) | `Disallow: /` |
| `Claude-User` | Anthropic | Page fetched for a Claude user | **No** (`false`) | `Disallow: /` |
| `PerplexityBot` | Perplexity | Perplexity search index | **No** (`false`) | `Disallow: /` |
| `Applebot-Extended` | Apple | Apple model training | **No** (`false`) | `Disallow: /` |
| `Meta-ExternalAgent` | Meta | Model training | **No** (`false`) | `Disallow: /` |
| `Amazonbot` | Amazon | Amazon services and Alexa answers | **No** (`false`) | `Disallow: /` |
| `Bytespider` | ByteDance | Model training | **No** (`false`) | `Disallow: /` |
| `CCBot` | Common Crawl | Open web archive for model training | **No** (`false`) | `Disallow: /` |

Anthropic's 3 bots, `PerplexityBot` and `CCBot` are explicitly blocked with `Disallow: /`, while OpenAI's `GPTBot`, `OAI-SearchBot` and `ChatGPT-User` are not named in the disallow group and inherit the open `User-agent: *` rules.

## How Site SEO & AI Crawler Audit compared with rivals

We ran the same `theguardian.com` check against three competing tools on 2026-10-20:

| Tool | Result on theguardian.com | Wall time | Score |
| --- | --- | --- | --- |
| **OpenKrill Site SEO & AI Crawler Audit** | All 14 crawlers with operator, purpose, allowed status and matching rule; follows 302 to `/uk`; reports `xRobotsTag` and `metaRobots` | 1,140 ms | 10.0 / 10 |
| **TechnicalSEO.com AI Bot Access Tool** | Browser run checked 35 AI user agents with matching `robots.txt` rule and live HTTP status (403 on most blocked bots), but required a browser session and did not label each bot's operator or purpose | 4,485 ms | 9.1 / 10 |
| **SiteAudit MCP** (`vdalhambra/siteaudit-mcp`) | `check_robots_txt` returned only the first 30 of 49 rules (all for `*`), cutting off the AI-bot disallow block completely so it could not tell whether `GPTBot` or `ClaudeBot` was blocked | 2,714 ms (2,146 ms startup + 568 ms call) | 5.2 / 10 |
| **Apify AI Crawler Policy Audit** | Not run: requires an Apify account and API token | not run | not run |

## Checking Open Graph tags and AI citation signals on the same page

Two additional calls on `https://www.theguardian.com/` show what else an agent can check in seconds:

- **Page tags and head size (`check_page_tags`).** The tool read the 48-character title (*Latest news, sport and opinion from the Guardian*), the 112-character description, the canonical link `https://www.theguardian.com/` and 6 icon links, and flagged `og:title`, `og:description`, `og:image`, `og:url` and `twitter:card` as missing in the first 150 KB of the HTML head (noting that tags placed after 150 KB of inline styles or scripts are not read).
- **AI readiness and citation checks (`ai_readiness_check`).** The domain serves `robots.txt` and `sitemap.xml` (286 entries read from `sitemaps/news.xml`), while `/llms.txt` and `/agents.md` return HTTP 404. Across the 5 AI citation signals on the home page, 1 passed (Q&A format, with 12 question headings), 1 warned (54 lists and 0 tables, but no Schema.org JSON-LD), and 3 failed (no summary block near the top, no author byline on the front page, and no `h1` heading).

## Connect it to your AI client

Add `https://sitecheck.openkrill.app/mcp` to your assistant, or follow the install steps for [ChatGPT](https://tools.openkrill.app/integrations/site-check/chatgpt), [Claude](https://tools.openkrill.app/integrations/site-check/claude), [Cursor](https://tools.openkrill.app/integrations/site-check/cursor) and [Pi](https://tools.openkrill.app/integrations/site-check/pi). The 3 agent audit workflows are downloadable at [site-check-jobs/SKILL.md](https://tools.openkrill.app/skills/site-check-jobs/SKILL.md).

Here is the JSON-RPC call from our crawler audit:

```
{"method":"tools/call","params":{"name":"check_ai_crawler_access",
 "arguments":{"url":"https://www.theguardian.com/"}}}
```

## Mistakes to avoid when auditing AI crawlers

- **Assuming one bot name per AI company.** OpenAI, Anthropic and Perplexity use separate user agents for model training (`GPTBot`, `ClaudeBot`), search indexing (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) and user-triggered page fetches (`ChatGPT-User`, `Claude-User`, `Perplexity-User`).
- **Confusing `robots.txt` with CDN firewall blocks.** `check_ai_crawler_access` reads the published `robots.txt` rules and page directives; a CDN or WAF can still block a bot by IP or user agent even when `robots.txt` allows it.
- **Burying Open Graph tags past 150 KB of inline CSS or state scripts.** Link-preview fetchers and `check_page_tags` stop reading oversized `<head>` blocks; keep meta and Open Graph tags near the top of `<head>`.

## Frequently asked questions

### Does my website's robots.txt block GPTBot, ClaudeBot or other AI crawlers?

Call check_ai_crawler_access on sitecheck.openkrill.app/mcp with your page URL. It tests 14 named AI crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended) and shows the exact robots.txt rule, X-Robots-Tag header and meta robots tag that apply to each one.

### How can I check which AI bots are allowed to crawl a site?

Paste the URL into tools.openkrill.app/site-check or call check_ai_crawler_access and ai_readiness_check on https://sitecheck.openkrill.app/mcp. You get a per-crawler table with the operator, purpose (training, search index or user fetch) and whether robots.txt allows it.

### Which free tool audits a page for AI search crawler access and SEO tags?

OpenKrill's Site SEO & AI Crawler Audit (tools.openkrill.app/tools/site-check) is free and needs no account or API key. It checks 14 AI crawlers in robots.txt, Open Graph and canonical tags, redirect chains, llms.txt and sitemap presence, and 5 AI citation signals.

### Why does check_page_tags say my Open Graph tags are missing when I can see them in DevTools?

Either the tags are injected by client-side JavaScript (which social unfurlers and AI crawlers do not run) or they sit after the first 150 KB of the HTML head. Move your title, description, canonical and og:* tags to the top of the server-rendered head.

## Sources

- Crawler access, page-tag and AI readiness runs: [Site SEO & AI Crawler Audit](https://tools.openkrill.app/tools/site-check) (`https://sitecheck.openkrill.app/mcp`), read on 2026-10-20 between 06:12 and 07:16 UTC.
- The Guardian robots.txt file: [theguardian.com/robots.txt](https://www.theguardian.com/robots.txt), read on 2026-10-20.
- Competitors tested: [TechnicalSEO.com AI Bot Access Testing Tool](https://technicalseo.com/tools/ai-bot-access/) and [vdalhambra/siteaudit-mcp](https://github.com/vdalhambra/siteaudit-mcp), tested on 2026-10-20.