When generative AI arrived, many site owners did the cautious thing and blocked every AI bot they could name. Website builders and security services made it a one-click setting, and in some cases switched it on by default.
Now many of the same businesses are asking why ChatGPT recommends their competitors and never them. Often the answer is in a file most of them have never opened. When we audited our own site this month, we found our CDN adding AI rules on top of our robots.txt that we never wrote. This guide explains what each AI crawler does, what blocking it really costs, and how to set rules that match what you want.
You can't be recommended by an AI assistant that isn't allowed to read your website.
The three kinds of AI crawlers
"AI crawler" is a catch-all. The major AI companies now run separate bots for separate jobs, and each one has its own name, or user agent, that you can allow or block on its own.
Three jobs, three different costs if you block them
Training
Collect pages that may be used to train future AI models.
- GPTBot
- ClaudeBot
- CCBot
- Google-Extended
- Applebot-Extended
- meta-externalagent
AI search
Build the indexes AI assistants search when they answer a question.
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
- Googlebot
- Bingbot
User fetch
Open a page in real time because a person asked about it or shared the link.
- ChatGPT-User
- Claude-User
- Perplexity-User
AI crawler list: who's who
Each company documents its bots: OpenAI, Anthropic, Perplexity and Google. Here's the short version.
| Bot (company) | What it does | If you block it |
|---|---|---|
GPTBotOpenAI | Collects content that may train OpenAI's models | Your pages aren't used for training |
OAI-SearchBotOpenAI | Finds pages to show and link in ChatGPT search | OpenAI says you won't appear in ChatGPT search answers |
ChatGPT-UserOpenAI | Visits a page when a ChatGPT user's request needs it | ChatGPT can't open your page for that user |
ClaudeBotAnthropic | Collects content that may train Claude models | Your pages aren't used for training |
Claude-SearchBotAnthropic | Improves the quality of Claude's search results | You may appear less in Claude's answers |
Claude-UserAnthropic | Fetches a page when a Claude user asks | Anthropic says this can reduce your visibility in user-directed search |
PerplexityBotPerplexity | Indexes pages for Perplexity's answers. Perplexity says it isn't used for model training | You can drop out of Perplexity answers |
Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | You disappear from Google. Never block it |
Google-Extended | Controls use of your content for Gemini training and grounding | No effect on Google Search, rankings or AI Overviews |
Applebot-ExtendedApple | Controls use of your content for Apple's AI training | Applebot still crawls for Siri and Spotlight |
CCBotCommon Crawl | Builds an open web archive that many AI models train on | Your pages stay out of future archive snapshots |
meta-externalagentMeta | Collects content for Meta's AI models and products | Your pages aren't used for Meta's AI |
Microsoft is the odd one out. Copilot draws on the Bing index, and there's no separate AI crawler to block, so blocking Bingbot removes you from Bing and weakens your presence in Copilot. Don't.
What blocking really costs you
Blocking search and user bots has an immediate cost. When someone asks ChatGPT, Claude or Perplexity for a recommendation, the assistant searches its index and may open pages live. If its bots can't reach your site, it has to rely on what other websites say about you, or it simply names a competitor whose site it could read.
Blocking training bots has a slower, less certain cost. Your content won't be used to train future models. That's a reasonable choice for publishers whose articles are the product. For a business that wants to be known, it may mean future models know less about what you do, though nobody can measure that effect precisely today.
Blocking Google-Extended costs you nothing in Google Search. Google says it doesn't affect inclusion or ranking in Search. AI Overviews and AI Mode are controlled by Googlebot, so blocking Google-Extended won't remove you from them. Google has also said it intends to support AIPREF, a shared machine-readable signal for AI training preferences, in early 2027.
What we recommend
Three robots.txt policies, and who each one suits
Allow everything
Search, user and training bots can all read your site. You get the most visibility in AI answers today and in future models.
Local services, clinics, firms, B2B, e-commerce
Search yes, training no
Allow search and user bots so you can be cited and linked, and block the training bots so your content isn't used to build models.
News sites, paid content, original research
Block all AI bots
No AI tool can read your site directly. AI answers about you rely on other websites, and competitors fill the gap.
Private or legally restricted content
robots.txt rules you can copy
Your robots.txt file lives at yourdomain.com/robots.txt. Each bot follows the group that names it most specifically, and only falls back to the User-agent: * group when no group names it.
Option 1: allow everything (recommended for most businesses)
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Option 2: allow AI search, block AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
In option 2, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot aren't named, so they follow the * group and are allowed. Changes can take a day or so to be picked up; OpenAI, for example, says its systems need about 24 hours to adjust after you update robots.txt.
Cloudflare's Content Signals Policy, published in September 2025, adds a line to robots.txt that states how your content may be used: search, ai-input (using it in AI answers) and ai-train. Our own file says Content-Signal: search=yes, ai-input=yes, ai-train=yes. These signals state a preference. They don't block anything, and it's up to each AI company whether to honour them.
Check your firewall, not only robots.txt
robots.txt is a polite request. The bigger risk for many businesses is a block they never chose, set at their host, CDN or security plugin, which turns bots away before they ever read robots.txt.
- Cloudflare: since July 1, 2025, new domains on Cloudflare block known AI crawlers by default. Cloudflare can also manage your robots.txt for you and add AI rules at the top of it. Look in your dashboard's AI Crawl Control and bot settings.
- Website builders: some, such as Squarespace, have a setting to block AI crawlers. Check your SEO or crawler settings.
- Security plugins and firewalls: broad rules against "bots" or unfamiliar user agents can catch AI crawlers, and occasionally Googlebot too.
That last kind of mistake is expensive. A firewall rule meant for AI scrapers that also blocks Googlebot removes you from Google Search, AI Overviews and AI Mode all at once.
How to test your setup in 10 minutes
- Open
yourdomain.com/robots.txtin a browser. Read every group and look for rules you didn't write. - Check your CDN or host dashboard for AI bot, bot fight or crawler settings, and note what each one is set to.
- Run a quick request as an AI bot, shown below. A
200response means the page was served. A403means something is blocking that user agent. - Look in your server or CDN logs for visits from GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Seeing them is the real proof they can reach you.
- Repeat after any change to your hosting, security plugins or CDN plan.
curl -I -A "OAI-SearchBot" https://yourdomain.com/
curl -I -A "ClaudeBot" https://yourdomain.com/
Treat the curl test as a quick signal, not proof. Some firewalls check a bot's IP address as well as its name, so a test from your own computer can be treated differently from the real crawler.
Bots that ignore the rules
The major AI companies say their crawlers follow robots.txt. User-triggered fetchers are a grey area: Perplexity says its Perplexity-User fetcher generally ignores robots.txt, and OpenAI says robots.txt rules may not apply to ChatGPT-User, because in both cases a person asked for the page. Some crawlers ignore the rules altogether, and Bytespider, operated by ByteDance, is the one most often reported. For those, robots.txt isn't enough and a firewall block is the right tool. On our own site, Bytespider is the only bot we block, and we block it at the CDN.
Mistakes to avoid
- Blocking every AI bot, then paying for "AI visibility" work that can't succeed
- Blocking GPTBot and assuming you're out of ChatGPT search, or the reverse. They are separate bots
- Blocking Google-Extended to stay out of AI Overviews. It doesn't do that, and blocking Googlebot would remove you from Google
- Editing robots.txt but never checking the CDN settings that override it
- Publishing an llms.txt file while the bots that might read it are blocked. Our guide to llms.txt explains what that file does and doesn't do
Once AI crawlers can reach you, measure what they send back. Our guide on tracking AI traffic in GA4 shows how, and our technical SEO checklist covers the rest of your crawl setup.
Is your site quietly blocking AI search?
Digital Relevance checks robots.txt, CDN and firewall settings as part of every AI Search engagement, then helps U.S. and Canadian businesses get cited and recommended by ChatGPT, Perplexity, Gemini and Google's AI features. Start with a free AI Search Snapshot, or see what our AI Search plan covers.
Get My Free AI Search SnapshotFrequently Asked Questions
Should I block GPTBot?
For most businesses, no. GPTBot collects content that may be used to train OpenAI's models, and allowing it means future models can learn about your business. Publishers whose content is their product may choose to block it. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot.
Does blocking AI crawlers remove me from ChatGPT?
Blocking OAI-SearchBot does: OpenAI says sites that block it won't appear in ChatGPT search answers. Blocking only GPTBot keeps your content out of training but still lets ChatGPT search find and cite your pages.
Does Google-Extended affect AI Overviews?
No. Google-Extended controls whether your content is used for Gemini training and grounding. Google says it doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode use Googlebot, which you should never block.
Do AI crawlers respect robots.txt?
The major AI companies, including OpenAI, Anthropic, Google and Perplexity, say their crawlers follow robots.txt. Fetchers triggered by a user's request may not, and some crawlers, such as Bytespider, are widely reported to ignore it. Use a firewall rule for bots that ignore robots.txt.
How do I know if Cloudflare is blocking AI crawlers?
Check AI Crawl Control and the bot settings in your Cloudflare dashboard, and look at your robots.txt for rules Cloudflare added. Since July 1, 2025, new Cloudflare domains block known AI crawlers by default. A request with an AI bot's user agent that returns 403 is another sign.