Site SEO & AI Crawler Audit in Pi

Questions it answers

Site SEO and AI Crawler Audit checks public web pages and reports what they publish right now: SEO tags, redirects, AI crawler rules, AI citation signals, landing page conversion and, for a site you own, what leaks publicly. People also call it Site Check.

The exposure check works only on a domain you prove you own: the first call gives a DNS TXT record or a file to publish, and the check runs once it is live. It never looks up a person, a user name or an email address.

Page and crawler checks: which of 14 AI crawlers robots.txt allows on a page and the rule that decided it; the title, description, canonical, Open Graph and Twitter tags and which are missing; and the 301 redirect chain with status codes and headers.

Store and AI checks: one product page's SEO and Product structured data (price, GTIN, availability); a domain's llms.txt, agents.md, UCP file, sitemap and AI crawler rules; five AI citation signals of one page, each with evidence and a fix; and a sample of up to 40 sitemap addresses for broken pages and redirects.

Landing page check: twelve conversion items such as the h1, the call to action and whether its links work, form fields, trust signals, a price section, viewport and a contact path, each with evidence and a fix. It is a checklist, not a forecast.

Exposure check: exposed or clean with the top 3 risks and fixes, from dangling subdomains, leaked files such as .env, admin paths and masked API keys in the page's scripts, plus six legal signals with official sources. No port scan, and not legal advice.

AI-built app check: for an app made with Lovable, Bolt, v0, Replit or any AI app builder, the missing security headers of any public app with the exact header lines for its host; and, for an app you prove you own, exposed files such as .env and .git, keys in the JavaScript it ships, and whether its Supabase tables or Firebase database, Firestore and storage rules let anyone read them. It reads only status codes and Supabase row counts, never a row's data, and never writes.

What it does not do: no speed test, no rankings or traffic, no JavaScript, no sign-in, no page text returned, and no private, local or internal addresses. It does not request a page that robots.txt disallows for Agent Tools. robots.txt is a request, not access control, and a firewall can treat real crawlers differently.

It keeps no account or profile of you. Answers are cached for five minutes, except owner checks (exposure_check, ai_app_check), which are never cached. The cleaned arguments of each call sit in a 30-day input log, and a daily count of calls per tool is kept.

Short questions this server answers, each with a direct answer: AI Crawler Access, Share Tags and Redirect Chain Check.

What it can do

• Shows whether robots.txt allows or blocks 14 published AI crawlers
• Shows the rule that decided each crawler, the sitemaps and any noindex or noai directives
• Reads a page's title, description, canonical link, language and icons
• Reads Open Graph and Twitter share tags and lists the common ones that are missing
• Follows the redirect chain with each status code and shows the final address and headers
• Checks one product page's title, description, h1, canonical link and Product structured data (price, GTIN, availability)
• Checks a domain for llms.txt, agents.md, the Shopify UCP file, a sitemap and AI crawler rules in robots.txt
• Checks a page for five AI citation signals: summary, author credentials, Q&A format, headings, structured data
• Checks up to 40 sitemap addresses for broken pages and redirect chains, skipping pages robots.txt disallows
• Checks a landing page's h1, call to action, form fields, trust signals, listed prices, viewport and contact path
• Tests up to five call-to-action links of a page and gives each finding with evidence and a fix
• Checks the exposed surface of a domain its owner proves: leaked files, dangling subdomains, default pages
• Finds API keys in the JavaScript an owned app ships, public source maps and open admin paths, keys masked
• Flags 6 legal signals on an owned app (COPPA, fonts, session replay, CAN-SPAM, renewals, DMCA) with sources
• Checks 8 security headers of any public app and gives the exact lines for Vercel, Netlify, Cloudflare and others
• Tells an owner whether anyone can read their Supabase tables or Firebase data, by status and row count only
• Refuses private, local and internal addresses and redirects into them

Tools

• check_ai_crawler_access, Check AI crawler access in robots.txt. Checks whether 14 AI crawlers may fetch a page. Use it when the user asks whether AI crawlers can read a website or page, for example "can AI crawlers read example.com?", "is GPTBot blocked in my robots.txt?" or "which AI bots are allowed on https://example.com/blog?". Pass the page address. Returns, for 14 AI crawlers, whether robots.txt allows the page and the rule that decided it, plus the sitemaps, the redirect chain, and the page's X-Robots-Tag header and robots meta tag. It requests the page as ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and a browser, and says if ChatGPT can read it in one verdict (readable, blocked by CDN or WAF, blocked by robots, needs JavaScript, redirect problem, needs sign-in) with the fix. That test is by user agent from Agent Tools servers, not from OpenAI's addresses. Do not use it for private or local addresses, for security scans, or to collect a site's content.
• check_page_tags, Check Open Graph and share tags. Reads title, description and share tags of a page. Use it when the user asks about the meta tags, canonical tag or Open Graph social preview of a web page, for example "what will the preview look like when I share this page?", "does my homepage have a meta description and canonical tag?" or "which Open Graph tags is my page missing?". Pass the page address. Returns the title, meta description, canonical link, language, viewport, icons, Open Graph (ogp.me) and Twitter Card tags, and a list of the common tags that are missing. Tags added by JavaScript are not seen. Do not use it to read a page's text or contact details, for private or local addresses, or for rankings and traffic.
• trace_redirects, Trace a 301 redirect chain. Traces the redirects and headers of a web address. Use it when the user asks where a web address redirects to, or for a 301 redirect chain, for example "why does http://example.com end up on another address?", "show the redirect chain for this link" or "does this site redirect http to https?". Pass the address. Returns each hop with its status code and target, the final address and status, whether http is upgraded to https, and the main response headers such as cache-control and strict-transport-security. At most 6 redirects are followed. Do not use it for private or local addresses, to read page content, or to scan a site for vulnerabilities.
• product_page_seo, Check product JSON-LD and page SEO. Checks the SEO and Product data of a product page. Use it when the user asks whether one product page of an online store is set up for search, for example "is https://shop.example.com/products/blue-mug set up for search?", "does my product page have Product structured data?" or "why does Google show no price for this product?". Pass the product page address. Returns title and meta description length, h1 headings, canonical link, Open Graph tags, and the Product structured data (JSON-LD): name, brand, SKU, GTIN, MPN, price, currency, availability, variant count and rating summary, plus a list of issues found. On Shopify it adds the product handle and whether a variant parameter is in the address. Works on any store. It reads one page, skips it if robots.txt disallows it for Agent Tools, and does not run JavaScript. Do not use it for page speed, rankings or traffic, to collect prices or catalogs, or for private or local addresses.
• ai_readiness_check, Check AI readiness of a store or page. Checks if AI agents can read and cite a store or page. Use it when the user asks if AI agents can read a store or cite a page, for example "does shop.example.com have an llms.txt?" or "will AI assistants cite this page?". Pass url for a page, or domain for domain files only. Returns whether it has /llms.txt (llmstxt.org), /agents.md, the UCP file at /.well-known/ucp (version, services) and /sitemap.xml, and which of 14 AI crawlers robots.txt (RFC 9309) blocks site-wide. With a page it also rates five citation signals (summary near the top, author credentials, Q&A format, heading structure, structured data) as pass, warn or fail with evidence and a fix. It requests only robots.txt, these files, the page and its Markdown copy, skips what robots.txt disallows for Agent Tools, and shows what is published, not whether an agent can buy. Do not use it for private or local addresses, to read page content, or to scan for vulnerabilities.
• link_sample_check, Check a sitemap sample for broken links. Finds broken or redirected pages in a sitemap sample. Use it when the user asks whether pages of an online store or website are broken or redirected, or for a broken link in the sitemap, for example "are any pages of shop.example.com broken?", "check my sitemap for 404s" or "do my product pages redirect?". Pass the domain. Reads the sitemap (for a sitemap index, the product, page, collection and blog sitemaps first) and checks up to 40 addresses spread across it, on that host only, returning counts by status, and for each broken, unreachable or redirected address its status and final address. It is a sample, not every page; it uses HEAD requests at low concurrency, skips addresses that robots.txt disallows for Agent Tools, and makes at most 46 requests. Do not use it for private or local addresses, to read page content, or to monitor a site over time.
• landing_page_check, Check a landing page for conversion. Checks a landing page for conversion problems. Use it when the user asks how to get more sign-ups, sales or enquiries from a landing page, for example "review https://example.com/ for conversion" or "does my landing page have a clear call to action?". Pass the page address. Returns twelve checks, each with a status, evidence and a fix: the h1 and the sentence under it, the primary call to action and its repeats, whether up to five call-to-action links work, form field count, trust signals, a visible price or next step, the viewport tag, HTML size, render-blocking hints and a contact path. It reads one page, tests a few links, skips what robots.txt disallows for Agent Tools, does not run JavaScript, and returns no page copy, reviews or contact details. It is a checklist, not a conversion forecast, a speed test or a design review. Do not use it for private or local addresses, rankings or traffic.
• exposure_check, Check the exposed surface of a site you own. Checks what a site or app the user owns leaks publicly. Use it when the user asks what an attacker could find on their own domain, host or app, for example "does my app leak a .env file?". Pass the domain, host or app page address, not a bare IP. The first call returns ownership_not_proven with a DNS TXT record or file: publish it, then call again with the same target. Then it gives exposed or clean, the top three risks with fixes, and evidence: dangling subdomains, leaked files (never their contents), masked API keys, source maps, admin paths, dev servers and default pages. It adds six legal signals with law and source (COPPA, Google Fonts, session replay, renewals, DMCA, CAN-SPAM) and shipNaked: 7 pre-launch pages (privacy, refund and more) with text to paste; not legal advice. GET only, at most 32 fixed paths; no crawl, sign-in or port scan. Do not use it on a target the user does not own, to look up a person, user name or email, or for private or local addresses.
• ai_app_check, Check an AI-built app for leaks and open data. Checks an app made with an AI app builder (Lovable, Bolt, v0, Replit, Cursor) for leaks and launch readiness. Use it when the user asks "is my Lovable app secure?", "is my Supabase data public?", "which security headers is my app missing?" or "is my app ready to launch?". For any public app it returns 8 security headers with the lines for the detected host, launchList: 20 of 160 launch items read from outside, each pass, fail or warn with a fix (checklist true adds the owner-only items), and shipNaked: 7 pre-launch pages (privacy policy naming each service, liability terms, cookie banner, deletion, refund line at checkout, legal contact, frontend keys) with text to paste. The first call also returns ownership_not_proven with a DNS TXT record or file: once published, the next call adds exposed files (.env, .git), masked keys in its JavaScript, source maps, open Supabase or Firebase data, and backend signals. GET only; database checks read status codes and row counts, never a row's data, and never write. Do not use it beyond the public checks on an app the user does not own, for private addresses, or to scan many sites.

Add Site SEO & AI Crawler Audit in Pi

Pi does not speak MCP on its own. The pi-mcp-adapter extension (these steps match version 2.34.0) loads remote servers and keeps their schemas out of the prompt until you ask. Install it once, then restart Pi:

pi install npm:pi-mcp-adapter

Add the server to a shared MCP file. Use .mcp.json in the project if only that project needs Site SEO & AI Crawler Audit, or ~/.config/mcp/mcp.json if you want it in every project. Pi also reads ~/.agents/mcp.json. A Pi-only override lives in ~/.pi/agent/mcp.json or .pi/mcp.json and wins over the shared file.

{
  "mcpServers": {
    "site-check": {
      "url": "https://sitecheck.openkrill.app/mcp",
      "lifecycle": "eager"
    }
  }
}

The url is enough. Do not set command: that field is for a local process. No API key header. lifecycle is eager so Pi connects at startup and can search tool names before the first call. A config that omits it still works: the first call connects. After restart, run /mcp and confirm the server is listed. Ask: Will AI assistants cite https://example.com/blog/guide? The adapter should offer check_ai_crawler_access, check_page_tags, trace_redirects, product_page_seo, ai_readiness_check, link_sample_check, landing_page_check, exposure_check, ai_app_check.

Leave directTools off unless you want those schemas in every turn. With a handful of tools the cost is small. The default proxy is the better fit: search for the tool, then call it. /mcp disable site-check writes a project override in .pi/mcp.json without editing the shared file. Run /reload after that change.

Pi will not find this server by scanning Cursor or Claude config unless you run /mcp setup and import it. Putting the url in .mcp.json is the direct path and the one this page tests against. The server is the same public endpoint the other clients use, with the same limits and the same privacy rules.

If it does not work

A call checked on 2026-10-17

example.com is a public page. The tool reads the head only. The request below was posted to https://sitecheck.openkrill.app/mcp as tools/call. HTTP 200. Source: the Site SEO & AI Crawler Audit server, read 2026-10-17.

{
  "method": "tools/call",
  "params": {
    "name": "check_page_tags",
    "arguments": {
      "url": "https://example.com"
    }
  }
}

• status. 200
• source. The website itself, requested from Agent Tools servers
• notice. This shows what the site publishes to a request from Agent Tools servers at this moment. Firewalls and CDNs can treat real crawlers or visitors differently, and robots.txt is a request that crawlers follow by choice, not
• url. https://example.com/
• finalUrl. https://example.com/

Repeat the call yourself if you need a newer reading. Cached answers expire. A rate limit is not a result: wait and try again. Nothing in the call is a ranking, a filing, or advice.

Using it in production

Nothing to install and nothing to run: no package, no Python or Node.js version and no environment variable. Site SEO & AI Crawler Audit runs on our servers, and your client only needs the URL https://sitecheck.openkrill.app/mcp.

Use it in other apps

• Site SEO & AI Crawler Audit in ChatGPT
• Site SEO & AI Crawler Audit in Claude
• Site SEO & AI Crawler Audit in Cursor
• Site SEO & AI Crawler Audit in VS Code

Roles that use Site SEO & AI Crawler Audit

• SEO specialist guide

Frequently asked questions

Words on this page

• MCP (Model Context Protocol) definition
• Streamable HTTP transport definition
• Tool primitive definition
• Rate limit policy definition

Source: plugin listing for site-check in this repository, and a live tools/call on 2026-10-17. As of 2026-10-17.