AI Training vs AI Citation: Getting the Crawler Decision Right

Most robots.txt guides published in 2023 and 2024 gave the same advice: block AI crawlers. Protect your content. Don't let the models scrape you.

That advice made sense at the time. AI companies were training on public web content at scale, and the mechanism for opting out was a single robots.txt directive. Blocking felt like the reasonable defensive move.

What those guides didn't anticipate—and most haven't updated to reflect—is that the landscape changed. You can opt out of being training data without opting out of being cited. You can block AI training and keep AI citations intact.

Most businesses don't know this. Most platforms don't give you the control to act on it, even if you do. And the guides that tell you to "block AI bots" are, in most cases, telling you to disappear from AI search results to solve a problem that didn't require that trade-off.

The decision most businesses get wrong

When businesses started worrying about AI training data in 2023, the common response was to add a blanket block in robots.txt. Block GPTBot. Block all AI crawlers. Done.

It felt protective. It was also, in many cases, self-defeating.

Now those same rules are blocking the citation crawlers that would have put those businesses in front of buyers actively asking AI engines for recommendations.

The issue isn't that businesses made a bad decision. It's that they made one decision where they should have made two.

Training crawlers vs citation crawlers

Every major AI company now operates two separate classes of web crawlers.

A training crawler collects your pages, your content, your writing—and that data is used to train future versions of the model. This is what people were worried about in 2023. It's a legitimate concern.

A citation crawler—sometimes called a search crawler or retrieval crawler—indexes your content so that when a user asks the AI a question, it can surface your page as a source. This is what makes you findable in ChatGPT Search, Claude, and Perplexity. It's the mechanism by which AI engines put your business in front of buyers.

Blocking a training crawler does not affect its corresponding citation crawler. They are separate systems with separate user-agent strings. robots.txt rules for one do not apply to the other. This is officially documented by OpenAI, Anthropic, and Google—and it is the single most important thing to understand about your AI crawler strategy today.

How each company has split their crawlers

OpenAI

GPTBot crawls the web to collect content that may be used for training OpenAI models. Blocking it is optional—it has no consequence for whether you appear in ChatGPT Search.

OAI-SearchBot builds the index that powers ChatGPT Search. If you block it, ChatGPT cannot surface your pages when users search for topics you cover. This one matters for visibility.

ChatGPT-User fetches specific pages during live conversations when a user shares a link or asks a question that requires reading a current page. Blocking it can stop ChatGPT reading your content in real time, although OpenAI notes that robots.txt rules may not apply to fetches a user asked for.

When GPTBot and OAI-SearchBot are both allowed, OpenAI may use a single crawl for both purposes, so you won't necessarily see two visits. Each one still follows its own robots.txt rule.

Anthropic

Anthropic formally documented its three separate crawlers in February 2026.

ClaudeBot collects content for AI model training. Blocking it opts you out of training—no other consequence.

Claude-SearchBot crawls the web to improve search result quality inside Claude. This is the crawler that determines whether your content surfaces when Claude users search for topics you cover. Anthropic's documentation is direct about the consequence of blocking it: doing so "may reduce your site's visibility and accuracy in user search results." For businesses that want visibility in Claude answers, Claude-SearchBot must be allowed.

Claude-User fetches specific pages during live Claude conversations. It operates when a Claude user asks a question requiring web access, or when Claude is verifying current information. Blocking it limits Claude's ability to read your content in real time.

Each of the three needs its own robots.txt entry. A site that blocked ClaudeBot thinking it had opted out of Claude altogether has only blocked training. Claude-SearchBot and Claude-User can still reach it.

Google, Perplexity, and the rest

Google works differently. Googlebot—the crawler that powers Search, including AI Overviews—should never be blocked. Google-Extended isn't a crawler at all, and you won't find it in your server logs: Google fetches every page with its normal crawlers. It's a robots.txt setting Google reads to decide whether content Googlebot has already crawled may be used to train Gemini and to ground Gemini's answers. Google says it doesn't affect inclusion in Google Search and isn't a ranking signal. So blocking it isn't purely a training decision. It's training and Gemini's answers, in one switch.

Perplexity currently runs two crawlers—PerplexityBot for indexing and Perplexity-User for live fetches. Neither is used for model training. Both affect whether your content appears in Perplexity answers, although Perplexity-User generally ignores robots.txt, because a person asked for the page.

What a considered robots.txt actually looks like

If your goal is to block training crawlers while staying citable in AI search, your robots.txt looks something like this:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Common Crawl: a public dataset many AI models train on
User-agent: CCBot
Disallow: /

# Also removes you from Gemini's answers, not just training
User-agent: Google-Extended
Disallow: /

# Apple's equivalent: opts out of Apple's AI training,
# not Siri or Spotlight search
User-agent: Applebot-Extended
Disallow: /

# Everyone else is allowed, including the citation crawlers:
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The citation crawlers aren't named, so they fall under the User-agent: * group and are allowed. You only need explicit rules for what you want to block. A crawler follows the group that names it and ignores the wildcard rules, so a specific rule always wins over User-agent: *.

One important caveat: robots.txt rules only work if crawlers can actually read the file. If Cloudflare's Bot Fight Mode is active—or if your host is blocking bots at the infrastructure level—crawlers may never reach your robots.txt at all, and none of these rules will apply. You can check whether AI crawlers can reach your site before making any of these changes.

What your platform decides for you

Here is where the conversation gets harder. Many businesses cannot execute this strategy because their platform doesn't give them access to robots.txt.

Squarespace doesn't let you edit robots.txt. It has one toggle, "Block known artificial intelligence crawlers", which blocks a fixed list of AI bots. Most are training crawlers such as GPTBot and ClaudeBot, and the main citation crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) aren't on it. But the list also includes Google-Extended and DuckAssistBot, so switching it on takes you out of Gemini's answers and DuckDuckGo's AI answers too. You can't change what's on the list.

GoHighLevel does offer a robots.txt editor under Settings → Domains → Manage. It is one of the more capable platforms in this regard. But no AI-specific guidance ships with it by default—you need to know which directives to add and why.

Kajabi generates robots.txt automatically, with no editor, and its help centre says not to use robots.txt for blocking. If you want AI bots blocked, Kajabi's security team can block them at the network level on request. There's no self-serve way to separate training crawlers from citation crawlers.

Wix has a robots.txt editor in its SEO settings, with no AI-specific controls. You write the directives yourself.

Webflow gives direct access to robots.txt in Site settings → SEO → Indexing. It can also send a Content-Signal header, a proposed standard for saying whether your content may be used for AI training, search or AI answers. This is one of the better options among hosted platforms.

WordPress (self-hosted) gives you full control. You can edit robots.txt directly or via an SEO plugin. This is the most flexible option, and why self-hosted WordPress remains the default recommendation for businesses where discoverability actually matters.

If you're on a platform that doesn't give you this control, the options are: find the workaround your platform supports, accept the limitation, or move to infrastructure you actually own.

The actual trade-off

The question "should I block AI crawlers?" has a better version: "which crawlers should I block, and for what reason?"

Training: optional. You can opt out without consequence to visibility.

Citation: not optional, if visibility is the goal. Blocking citation crawlers removes you from AI search answers—the very thing most businesses are now trying to get into.

Most businesses haven't separated these two questions. Most guides haven't helped them do so. And most platforms haven't given them the tools to act on the distinction even when they understand it.

That's the gap. And it's why businesses that appear in AI answers aren't necessarily better—they just made the distinction at some point, intentionally or by accident.

Aimee Q Devlin is a Systems and Infrastructure Architect based in San Miguel de Allende, Mexico. She works with founders and operators of established businesses who are ready to rebuild their systems properly—including the infrastructure that makes those systems discoverable. The PRISM Diagnostic is where engagements begin.

→ How to Get Cited by ChatGPT, Perplexity, and Claude

→ What Is llms.txt—and Should Your Website Have One?

→ What is Discoverability?

→ The PRISM Diagnostic