Does the New York Times block GPTBot and other AI crawlers?
Yes. The robots.txt of www.nytimes.com blocks 13 of the 14 AI crawlers the tool checks: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot. It lets Amazonbot read the site.
| Page status | HTTP 200 |
|---|---|
| robots.txt | found, HTTP 200 |
| Sitemaps in robots.txt | https://www.nytimes.com/sitemaps/new/news.xml.gz, https://www.nytimes.com/sitemaps/new/sitemap.xml.gz and 3 more |
| Amazonbot (Amazon, Amazon services and Alexa answers) | allowed, no rule names it |
| GPTBot (OpenAI, model training) | blocked by Disallow: / |
| OAI-SearchBot (OpenAI, ChatGPT search) | blocked by Disallow: / |
| ChatGPT-User (OpenAI, page fetched for a ChatGPT user) | blocked by Disallow: / |
| Blocked crawlers | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot |
Frequently asked questions
Is GPTBot blocked in my robots.txt?
check_ai_crawler_access says, for each of 14 AI crawlers, whether robots.txt lets it fetch the page and which rule decided it.
It also returns the sitemaps, the redirect chain, the X-Robots-Tag header and the robots meta tag. It reads robots.txt only, so it cannot tell whether a firewall or bot filter blocks a crawler. Tool: check_ai_crawler_access on https://sitecheck.openkrill.app/mcp.
What will my link preview look like when shared?
check_page_tags returns the page title, meta description, canonical link and the Open Graph and Twitter tags, plus a list of the common tags that are missing.
Tags that JavaScript adds after the page loads are not seen. Tool: check_page_tags on https://sitecheck.openkrill.app/mcp.
Show the redirect chain for a URL?
trace_redirects lists each hop with its status code and target, the final address, and whether http is upgraded to https.
It follows at most 6 redirects and also returns main headers such as cache-control and strict-transport-security. Tool: trace_redirects on https://sitecheck.openkrill.app/mcp.
Will AI assistants cite this page?
ai_readiness_check rates five citation signals on a page as pass, warn or fail, with the evidence and a fix for each.
The signals are a summary near the top, author credentials, a question and answer format, heading structure and structured data. For the domain it also checks /llms.txt, /agents.md, the sitemap and which AI crawlers robots.txt blocks site-wide. Tool: ai_readiness_check on https://sitecheck.openkrill.app/mcp.
Does my product page have Product structured data?
product_page_seo reads one product page and returns its Product JSON-LD, such as name, brand, SKU, price, currency and availability, with a list of issues found.
It also checks the title and description length, the h1 headings, the canonical link and the Open Graph tags. It works on any store, does not run JavaScript and skips a page that robots.txt closes to Agent Tools. Tool: product_page_seo on https://sitecheck.openkrill.app/mcp.
Run it yourself
The answer at the top of this page comes from a real call on 2026-10-24. Call the tool on the server for current data. The server is Site SEO & AI Crawler Audit at https://sitecheck.openkrill.app/mcp, with no key. Install steps: ChatGPT, Claude, Cursor, VS Code, Pi. The full tool list is in the install guides.