AI crawler checker: which AI bots can read your site?

Enter a domain. We read its robots.txt the way crawlers do and show, for 12 AI bots, whether the home page (or any path you choose) is allowed or blocked, and which rule decides it. We also check for /llms.txt.

How the check works

Our server downloads https://your-domain/robots.txt (up to 500 KB, following at most three redirects) and applies the matching rules from RFC 9309, the robots.txt standard that Google also documents in its robots.txt guide:

• Group selection. A bot obeys the group whose User-agent line names it (case-insensitive). If several groups name it, their rules are combined. Only when no group names the bot does it fall back to User-agent: *. That is why a site can block everything for * and still allow GPTBot, or the reverse.
• Longest match wins. Inside the group, the Allow or Disallow rule with the longest matching path decides. When an Allow and a Disallow are equally long, Allow wins. Wildcards * and the end anchor $ are supported.
• Missing or broken file. A 4xx response (for example 404) means "no restrictions". A 5xx error or an unreachable server means crawlers should assume the whole site is disallowed, so we show every bot as blocked in that case.

The table shows the rule that decided each result and its line number, so you can find it in your file in seconds.

Training bots, search bots and user agents

AI companies now run several crawlers each, and blocking one does not block the others:

• Training: collect pages that may be used to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose public dataset is widely used for AI research) and Bytespider (ByteDance). Google-Extended (Google) and Applebot-Extended (Apple) are not separate crawlers: they are control tokens that tell Google and Apple whether content their regular bots fetch may be used for their AI models. Google states that Google-Extended does not affect inclusion in Google Search.
• Search: build the index that AI search answers cite with links. OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot. Blocking these can keep your pages out of those answers.
• User-triggered: fetch a page because a person asked the assistant to open it. ChatGPT-User, Claude-User and Perplexity-User. Some operators say such fetches may not follow robots.txt in the same way, so read their documentation linked above.

A common setup for a small site that wants AI visibility but not model training: block the training bots, allow the search and user bots. Our guide how to block AI crawlers with robots.txt has copy-paste rules for each case, and the AI search hub explains how pages end up cited in ChatGPT, Perplexity and Google AI answers.

What the result does not tell you

robots.txt is a request, not a lock: well-behaved bots follow it, others may not. The tool also does not see firewall or CDN rules (for example a bot-blocking setting at your host) that can block a bot even when robots.txt allows it. If a bot is "allowed" here but never visits, check your server logs and CDN settings.

And llms.txt?

We also check whether /llms.txt exists. It is a community proposal for a short Markdown map of your site for language models, not part of robots.txt and not a standard that bots are required to read. If you want one, our llms.txt generator drafts it from your sitemap.

FAQ

Does blocking GPTBot remove my site from ChatGPT search?

Not by itself. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate bots. To stay visible in ChatGPT search while opting out of training, block GPTBot and allow OAI-SearchBot.

Will blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether content may be used for Gemini models and grounding.

Why does a bot show as allowed when I added a rule for it?

Usually the rule is in a group with a typo in the User-agent name, or a longer Allow path overrides your Disallow. The table shows which group and line decided the result.

My robots.txt returns an error. What do bots do?

Under RFC 9309, a 4xx (like 404) means crawlers may access everything; a 5xx server error means they should treat the whole site as disallowed until the file is reachable again.

Is this check stored or shared?

Results are cached for one hour so repeated checks are fast. We do not need an account or email, and we only fetch robots.txt and llms.txt from the domain you enter.

Other free tools: llms.txt generator · Sponsored link checker · All tools

Updated: 2026-10-16