How to Block AI Crawlers in robots.txt and Still Show Up in AI Search

To block AI crawlers in robots.txt without disappearing from AI search, disallow the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) alone. Most vendors now run separate user agents for training, for search indexing and for live fetches a user asks for, and each one is controlled separately. Below is a bot-by-bot table built only from the vendors' own documentation, three ready-to-paste robots.txt files, and a way to check that the rules actually work.

Published: 2026-10-16 · 9 min read · Satelo

Three kinds of AI bots, three different decisions

"AI crawler" used to mean one thing: a bot collecting text to train a model. That is no longer true. The big vendors now split their traffic into three roles, and blocking each role costs you something different.

• Training crawlers collect pages that may be used to train future models. Blocking them means your content should stay out of future training data. You lose nothing in search.
• Search crawlers build the index that AI search answers draw from, and those answers link back to the source. Blocking them means you stop being shown or cited as a source in that product.
• User-triggered fetchers visit a single page because a person asked the assistant about it or pasted the link. They don't crawl the web on their own, and several vendors say robots.txt may not apply to them.

The common mistake is treating all three as one. A blanket "block all AI" rule usually hits the search crawlers too, and that removes you from exactly the answers you might want to be cited in. If AI visibility matters to you, read our guide on how to get cited by ChatGPT first, then come back and decide what to block.

The bot table: what each user agent does and what blocking costs

Every token below was checked against the vendor's own page on October 16, 2026. Tokens are case-insensitive in robots.txt, but copy them exactly as shown anyway.

Sources, each opened on the day of writing: OpenAI crawlers, Google common crawlers, Anthropic crawlers, Perplexity crawlers, About Applebot, Common Crawl FAQ.

One detail from OpenAI's page matters for planning: the settings are independent, and if you allow both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes. So "allow search, block training" is a real, supported setup, not a loophole.

Copy-paste file 1: block training, keep AI search

This is the setup most content sites are looking for. Training bots get a full disallow. Search bots, user fetchers and ordinary search engines are untouched.

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Googlebot and Bingbot
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Several User-agent lines on top of one rule set is valid syntax; Google's robots.txt documentation says you can group rules for several crawlers "by repeating user-agent lines". Replace the sitemap URL with your own.

Copy-paste file 2: block every AI bot, search included

Use this only if you've decided you don't want to appear in ChatGPT search, Claude or Perplexity at all. It keeps classic search engines working.

# Training, AI search and user-triggered AI fetchers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Be honest with yourself about the trade: this file asks AI assistants not to read or cite you, but it can't stop them from talking about your brand using other sources. And as the next sections show, the user-triggered fetchers may ignore it.

Copy-paste file 3: allow everything, protect only private paths

If you want maximum visibility, the simplest file is also the safest. This is roughly what we run on satelo.io (more on that below).

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

The merge trap: a bot obeys only one group

Most broken AI rules come from editing an existing file badly. Google's robots.txt specification says "only one group is valid for a particular crawler": the crawler picks the group with the most specific matching user agent and ignores the rest. Two consequences:

• Named groups don't inherit from *. If your User-agent: * group hides /admin/ and /cart/, and you add a separate User-agent: OAI-SearchBot group with only Allow: /, OAI-SearchBot will now crawl /admin/ and /cart/. Copy your private-path rules into every named group.
• A blanket disallow catches search bots. User-agent: * with Disallow: / blocks OAI-SearchBot, Claude-SearchBot and PerplexityBot too, unless each has its own allowing group. We cover this case in how to get your website indexed by ChatGPT.

Also remember that robots.txt works per host. Anthropic's page explicitly says to add the rules to each subdomain you want to opt out, and the same logic applies to every vendor: blog.example.com/robots.txt is a separate file from example.com/robots.txt.

Google is a special case: there is no "AI Overviews" bot

People often ask for a token that blocks Google's AI Overviews but keeps normal search. There isn't one. Google's AI features documentation says AI Overviews and AI Mode rely on the regular Search crawl, so they're controlled through Googlebot and the usual preview controls: nosnippet, data-nosnippet, max-snippet and noindex.

• Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI. It's not a Search switch.
• Blocking Googlebot removes you from Google Search crawling as a whole, not just from AI features. Almost no site should do that.
• Snippet controls limit how much text Google can show, in classic results and in AI features alike.

If showing up in those summaries is the goal rather than the worry, see how brand mentions work in AI Overviews.

robots.txt is a request, not a lock

Google says it plainly in its introduction to robots.txt: the file "cannot enforce crawler behavior", and while reputable crawlers obey it, "other crawlers might not". For AI bots, that means:

• User-triggered fetchers sit in a gray zone. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Anthropic's page says its bots honor robots.txt and lists Claude-User among them.
• Anyone can fake a user agent. A scraper can call itself GPTBot. That's why OpenAI publishes IP ranges for its bots (linked from its crawler page) and why log checks should look at IPs, not just names.
• It only affects future visits. The rules tell bots what to do from now on. None of the vendor pages above promise to delete what was collected before you added the block, so don't expect that.
• Changes take time. OpenAI and Perplexity both mention roughly 24 hours before their systems pick up a robots.txt change, and Google says it generally caches robots.txt for up to 24 hours.

If content truly must not be read by machines, put it behind a login. robots.txt is for polite bots.

Check your CDN and firewall: the silent block

Your robots.txt can be perfect while your CDN blocks the bots anyway. Many hosts and CDNs now offer one-click AI bot blocking. Cloudflare, for example, lets you allow or block individual AI crawlers in AI Crawl Control. These switches act before robots.txt is ever read, and some cover search and assistant bots, not just training bots.

Checklist, five minutes:

1. Open your CDN or WAF dashboard and look for "AI bots", "AI crawlers" or "bot fight" settings.
2. List exactly which bots each setting blocks. Make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed if you want AI search traffic.
3. Check rate limits and country blocks. A crawler that keeps getting 403 or 429 responses is effectively blocked.
4. Look at security plugins too. Some WordPress firewall plugins block unfamiliar user agents by default.

How to verify it's working

Don't trust the file, trust the logs. Three checks:

1. Open your live file at https://yourdomain.com/robots.txt in a private window. Caching plugins and CDNs sometimes serve an old copy.
2. Grep your access logs for the tokens and look at the status codes you return:

   grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider" access.log \
     | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -30

   The first column is the hit count, then IP and HTTP status. Blocked training bots should mostly request only /robots.txt after the change. Search bots should get 200s.
3. Match IPs for anything surprising against the vendor's published ranges. A "GPTBot" from an IP not on OpenAI's list isn't OpenAI.

The log format above is the common Apache/Nginx combined format; adjust the column numbers if yours differs.

What we do on satelo.io, and why

Our own robots.txt allows every bot and disallows only /api/. That is a deliberate choice for a small site in public beta. We want people asking AI assistants about content exchanges, guest posts or site quality to have a chance of seeing our guides, and we have no paywalled archive whose value would leak into training data. The /api/ paths are endpoints, not pages, so there is nothing there for any crawler to index.

We also publish an llms.txt file, with no illusion that it's a ranking factor; it's just a cheap, readable map. Your situation may differ. A publisher whose business is original reporting has a strong case for file 1. A local business or a young blog usually gains more from being readable everywhere.

What to do next

1. Decide per role: training, search, user fetch. Write the decision down in one sentence.
2. Paste file 1, 2 or 3, copying your private-path rules into every named group.
3. Check your CDN, WAF and security plugins for AI bot switches.
4. Recheck your logs after a few days.
5. If you want AI to mention you, blocking is only half the job; see how to get mentioned by AI.

If you build AI agents yourself, there's a fairer path than scraping: talk to sites through APIs that their owners opted into. Satelo's MCP server for AI agents lets an agent check any domain's Quality Score with the evidence behind each number, find sites in the network and handle article exchanges through official tools, with no scraping involved. And if you just want to know how your own site looks from the outside, run the free check; it's built mostly from open data and takes about 30 seconds.

FAQ

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, and the two settings are independent. Block GPTBot and allow OAI-SearchBot to stay out of training while remaining eligible for ChatGPT search answers.

How do I block AI crawlers but not Google?

Disallow the AI tokens by name (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and so on) and leave Googlebot and the User-agent: * group open. Google says Google-Extended does not affect inclusion or ranking in Google Search.

Can I block Google AI Overviews with robots.txt?

Not separately. Google's AI features use the normal Search crawl, so the only robots.txt control is Googlebot, which would remove you from Search too. Use nosnippet, data-nosnippet or max-snippet to limit what text Google can show.

Do AI crawlers actually respect robots.txt?

The documented training and search crawlers from OpenAI, Google, Anthropic, Perplexity, Apple and Common Crawl say they follow it. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. robots.txt can't enforce anything; use logins for content that must stay private.

What is the difference between ClaudeBot and Claude-SearchBot?

Anthropic describes ClaudeBot as the crawler that collects content that may be used for model training, and Claude-SearchBot as the one that indexes content to improve Claude's search results. Blocking ClaudeBot opts you out of training; blocking Claude-SearchBot may reduce your visibility in Claude's answers.

How long does a robots.txt change take to work?

OpenAI and Perplexity both mention about 24 hours for their systems to adjust, and Google says it generally caches robots.txt for up to 24 hours. Check your server logs a few days after the change.

Does blocking AI bots delete content they already collected?

Don't count on it. robots.txt tells bots what to do from now on, and none of the vendor pages cited here promise to remove previously collected content when you add a block.

Read next

Generative engine optimization: how to get cited in AI answersGenerative engine optimization (GEO) without hype: where the term comes from, what Google, OpenAI and Yandex say about AI answers, plus a page checklist.

GEO vs SEO vs AEO: What Actually Changes for a Small WebsiteGEO vs SEO vs AEO explained: where the terms came from, what Google and OpenAI officially say, what changes for a small site, and a 30-day plan to act on it.

How to Get Your Website Into ChatGPT Search: OAI-SearchBot, Bing and MentionsHow to get your website indexed by ChatGPT: allow OAI-SearchBot, unblock OpenAI at your CDN, get into Bing, earn mentions and check server logs to confirm.

How to Rank in Microsoft Copilot: Start With the Bing IndexHow to rank in Microsoft Copilot: get indexed by Bing, set up Bing Webmaster Tools, ping IndexNow, fix meta tags that block citations, and track citations.