How we picked the sites
We took every domain with "cleaning" in its name from the Tranco top 1 million list, a public ranking of busy domains. We left out names that looked like registries, forums, supply shops or non-English markets, which gave 28 domains. From the page titles, we then dropped 10 more: 1 software seller, 2 product makers, 1 supply shop, 1 review blog, 1 empty blog, 2 non-English sites, 1 gambling page and 1 basketball site. That left 18 cleaning firms that sell a service. We do not name any of the 18 here.
For each site we called 2 Site Check tools. check_ai_crawler_access reads robots.txt, headers and redirects for 14 named AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot. check_page_tags reads the page tags. We counted 8 of them: meta description, canonical link, mobile viewport, og:title, og:description, og:image, og:url and twitter:card.
What we found
| Result | Sites |
|---|---|
| Home page sent back to us | 9 of 18 |
| Refused our server (403) | 4 of 18 |
| CDN error or no answer | 5 of 18 |
| Blocks AI crawlers in robots.txt | 1 of 9 |
| All 8 counted tags present | 2 of 9 |
No share image (og:image) | 7 of 9 |
| No share title and description | 5 of 9 |
| No canonical link | 3 of 9 |
| No meta description | 2 of 9 |
| No mobile viewport tag | 2 of 9 |
The one site that blocks AI crawlers shuts out 12 of the 14 we check: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot. It still lets in the other 2 we check, Bytespider and CCBot. That covers both the bots that collect training data and the ones that fetch a page when a person asks an assistant a question (OpenAI, Overview of OpenAI crawlers). That can be a choice. If it is not your choice, it is a change in robots.txt (RFC 9309, Robots Exclusion Protocol).
The 9 sites that sent nothing back
A 403 is a refusal of our server. A CDN error or no answer means the site did not serve us, and we cannot tell why. Neither tells us what GPTBot or ClaudeBot gets, because a site can treat each bot by its own rule. So we do not count those 9 as blocked. The reverse is true too: a firewall or CDN rule can turn a bot away even when robots.txt lets it in, and a robots.txt check cannot see that.
What to check on your own site
Open your robots.txt and look for Disallow: / under a bot name you did not mean to block. Then check your home page for a title, a meta description and a canonical link, which tell search engines what the page is and which address is the real one. Add og:title, og:description and og:image too: they decide what people see when someone shares your link (The Open Graph protocol).
None of this makes an AI assistant name you. Nobody can promise that. It only makes sure the door is not shut by mistake.
Run the same check on your site: Site Check is free, with no sign-up and no key. Open Site Check, or add it in Claude, ChatGPT or Cursor, then ask "can AI crawlers read example.com?". If you would rather get a written report with the fixes in order, we do that as a fixed-price AI search readiness audit.
Sources
- Tranco list: a research ranking of the top 1 million domains, downloaded 2026-10-07.
- Koster, Illyes, Zeller, Sassman. RFC 9309, Robots Exclusion Protocol (2022): how robots.txt groups and Disallow rules work.
- OpenAI, Overview of OpenAI crawlers: GPTBot collects training data, OAI-SearchBot and ChatGPT-User fetch pages for search and for a user's question.
- The Open Graph protocol: the og:title, og:description and og:image tags a link preview reads.
- Our own runs: Site Check
check_ai_crawler_accessandcheck_page_tagson 18 cleaning-firm sites, 2026-10-10 about 22:00 GMT+7.