[Agent Tools](https://tools.openkrill.app/) / [Blog](https://tools.openkrill.app/blog)

# Can AI crawlers read cleaning-company websites? We checked 18, and 9 answered

On the evening of 2026-10-10 we ran 18 cleaning-company websites through our free Site Check tool. 9 of them sent their home page back to us, and only 1 of those 9 blocks AI crawlers in its robots.txt. The most common gap was a missing share image, on 7 of the 9.

By OpenKrill. Published 2026-10-11. Every number here comes from our own Site Check runs on 2026-10-10 at about 22:00 GMT+7. 9 sites read once is a small snapshot, not a survey, so read the numbers that way.

## How we picked the sites

We took every domain with "cleaning" in its name from the [Tranco top 1 million list](https://tranco-list.eu/), a public ranking of busy domains. We left out names that looked like registries, forums, supply shops or non-English markets, which gave 28 domains. From the page titles, we then dropped 10 more: 1 software seller, 2 product makers, 1 supply shop, 1 review blog, 1 empty blog, 2 non-English sites, 1 gambling page and 1 basketball site. That left 18 cleaning firms that sell a service. We do not name any of the 18 here.

For each site we called 2 Site Check tools. `check_ai_crawler_access` reads robots.txt, headers and redirects for 14 named AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot. `check_page_tags` reads the page tags. We counted 8 of them: meta description, canonical link, mobile viewport, `og:title`, `og:description`, `og:image`, `og:url` and `twitter:card`.

## What we found

| Result | Sites |
| --- | --- |
| Home page sent back to us | 9 of 18 |
| Refused our server (403) | 4 of 18 |
| CDN error or no answer | 5 of 18 |
| Blocks AI crawlers in robots.txt | 1 of 9 |
| All 8 counted tags present | 2 of 9 |
| No share image (`og:image`) | 7 of 9 |
| No share title and description | 5 of 9 |
| No canonical link | 3 of 9 |
| No meta description | 2 of 9 |
| No mobile viewport tag | 2 of 9 |

The one site that blocks AI crawlers shuts out 12 of the 14 we check: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot. It still lets in the other 2 we check, Bytespider and CCBot. That covers both the bots that collect training data and the ones that fetch a page when a person asks an assistant a question ([OpenAI, Overview of OpenAI crawlers](https://platform.openai.com/docs/bots)). That can be a choice. If it is not your choice, it is a change in robots.txt ([RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)).

## The 9 sites that sent nothing back

A 403 is a refusal of our server. A CDN error or no answer means the site did not serve us, and we cannot tell why. Neither tells us what GPTBot or ClaudeBot gets, because a site can treat each bot by its own rule. So we do not count those 9 as blocked. The reverse is true too: a firewall or CDN rule can turn a bot away even when robots.txt lets it in, and a robots.txt check cannot see that.

## What to check on your own site

Open your robots.txt and look for `Disallow: /` under a bot name you did not mean to block. Then check your home page for a title, a meta description and a canonical link, which tell search engines what the page is and which address is the real one. Add `og:title`, `og:description` and `og:image` too: they decide what people see when someone shares your link ([The Open Graph protocol](https://ogp.me/)).

None of this makes an AI assistant name you. Nobody can promise that. It only makes sure the door is not shut by mistake.

**Run the same check on your site:** Site Check is free, with no sign-up and no key. [Open Site Check](https://tools.openkrill.app/site-check), or add it in [Claude](https://tools.openkrill.app/integrations/site-check/claude), [ChatGPT](https://tools.openkrill.app/integrations/site-check/chatgpt) or [Cursor](https://tools.openkrill.app/integrations/site-check/cursor), then ask "can AI crawlers read example.com?". If you would rather get a written report with the fixes in order, we do that as a fixed-price [AI search readiness audit](https://tools.openkrill.app/aiready).

## Sources

- [Tranco list](https://tranco-list.eu/): a research ranking of the top 1 million domains, downloaded 2026-10-07.
- [Koster, Illyes, Zeller, Sassman. RFC 9309, Robots Exclusion Protocol (2022)](https://www.rfc-editor.org/rfc/rfc9309): how robots.txt groups and Disallow rules work.
- [OpenAI, Overview of OpenAI crawlers](https://platform.openai.com/docs/bots): GPTBot collects training data, OAI-SearchBot and ChatGPT-User fetch pages for search and for a user's question.
- [The Open Graph protocol](https://ogp.me/): the og:title, og:description and og:image tags a link preview reads.
- Our own runs: Site Check `check_ai_crawler_access` and `check_page_tags` on 18 cleaning-firm sites, 2026-10-10 about 22:00 GMT+7.