AI Visibility Turkey
TR Request audit

Technical guide

AI Crawlers and robots.txt: Which Bots to Allow

Each AI company runs more than one bot, and some of them decide whether you appear in answers while others concern model training. Deciding those two things separately in your robots.txt is how you avoid switching your visibility off by accident.

AI Visibility Turkey editorial team Published: Last updated: 8 min read
On this page
  1. Two decisions: appearing and training
  2. Bots engine by engine
  3. Decision table
  4. Sample robots.txt
  5. CDN and firewall note
  6. What robots.txt cannot do

In short

  • Appearing in answers (search bots) and allowing model training (training bots) are separate settings; do not manage them as if they were one decision.
  • For user-triggered bots (ChatGPT-User, Perplexity-User), robots.txt may not apply; the providers say so in their own documentation.
  • According to OpenAI, robots.txt changes can take about 24 hours to take effect; blocks at server or CDN level work independently of robots.txt.

Two decisions: appearing and training

There used to be one question for a site: should Googlebot crawl it? Today each AI company runs several bots for different purposes. The distinction shared across the providers' own documents is this: some bots index pages so they can show your site when a user asks a question; some collect content to train the model itself; and some run only when a user asks for your page to be opened.

A Disallow line written without knowing this can switch off your visibility by accident. Conversely, "let everyone in" is also a business decision about whether your content may be used in model training. The right answer depends on the business: a hospital publishing patient information, a manufacturer with an export catalog and a publisher selling paid content may not choose the same policy.

Bots engine by engine

Bot groups per each provider’s own documentation. The source is linked on each card.

Search, training and user bot

OpenAI (ChatGPT)

Documented by the engine

If you want to appear in ChatGPT search you need to allow OAI-SearchBot. According to OpenAI, sites that opt out of this bot do not appear in ChatGPT search answers (they can still show as navigational links).

  • OAI-SearchBot: used to surface sites in ChatGPT search
  • GPTBot: may be used for model training; a setting independent of search
  • ChatGPT-User: user-initiated visits; robots.txt may not apply

OpenAI also advises allowing the IP ranges it publishes for OAI-SearchBot; changes take about 24 hours.

Source: OpenAI: Bots ↗

Search and user bot

Perplexity

Documented by the engine

Perplexity’s page explains bot behavior but does not explain how sources are selected or ranked; that information is not published.

  • PerplexityBot: surfaces and links sites in Perplexity search; Perplexity says it is not used for foundation-model training
  • Perplexity-User: fetches pages on user request; generally ignores robots.txt

Source: Perplexity: Bots ↗

Training, search and user bot

Anthropic (Claude)

Documented by the engine

According to Anthropic, blocking the last two bots may reduce your visibility in Claude’s search answers.

  • ClaudeBot: collects training data
  • Claude-User: fetches pages when a user’s question needs it
  • Claude-SearchBot: indexes pages for search relevance

Anthropic advises against blocking by IP address; it publishes crawler IP ranges at claude.com/crawling/bots.json.

Source: Anthropic: Does Anthropic crawl data from the web ↗

Search and training permission

Google (Gemini and AI features)

Documented by the engine

Google-Extended is a robots.txt token separate from Googlebot. Blocking it does not change where your pages appear in Google Search.

  • Googlebot: Google Search index; for AI Overviews and AI Mode a page must be indexed
  • Google-Extended: robots.txt token for Gemini training and grounding; does not affect inclusion in Search

Source: Google: Common crawlers ↗

We do not give a bot card for Microsoft Copilot, Grok or DeepSeek in this guide: on the Microsoft side, Bing Webmaster Tools' "AI Performance" report shows citation counts (counts only, not placement); xAI’s documentation describes Grok’s web search as running live, with the model deciding on inline citations; and for DeepSeek we found no primary documentation on source selection. We do not present undocumented things as documented.

Decision table: what must be open to appear?

Scroll the table sideways.

robots.txt decision by goal
Your goalBot to allowWhat happens if you block it
Appear in ChatGPT searchOAI-SearchBotAccording to OpenAI you do not appear in ChatGPT search answers (you may show as a navigational link).
Appear in Perplexity answersPerplexityBotYour chance of being surfaced and linked in Perplexity search drops.
Appear in Claude search answersClaude-SearchBot and Claude-UserAccording to Anthropic your visibility in Claude’s search answers may drop.
Appear in Google AI Overviews and AI ModeGooglebot (page must be indexed and snippet-eligible)A page that is not indexed cannot be a supporting link in these features.
Allow model trainingGPTBot, ClaudeBot, Google-ExtendedTraining permission is off; Google-Extended does not affect Search inclusion.

Source: OpenAI, Perplexity, Anthropic and Google bot documentation

Which policy fits whom?

  • Maximum visibility (allow everything): the default for service providers, exporters, clinics and online stores whose goal is reaching new buyers. Choose it knowing that it also grants training permission.
  • Search yes, training no: for publishers with original research, paid content or copyright sensitivity. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot open; GPTBot, ClaudeBot and Google-Extended closed. The price: your content is not part of those bots' training data.
  • Everything closed: can be chosen on privacy or licensing grounds, but it means giving up any expectation of visibility in AI answers.

This three-way split is our own framework, not a recommendation from the providers' documents. The choice is a business and legal decision.

Sample robots.txt

The file below reflects the policy we use on our own site: every search and training bot is allowed. Replace yourdomain.com with your own domain.

# Visibility-first policy: everyone is welcome, including AI search and training bots
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

An example alternative if you want to allow search bots and switch training bots off:

# Alternative: allow search and citation bots, switch training permissions off
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The second file is an example only; review it with your technical team, and with legal counsel if needed, before applying it. Bot names must match exactly, and providers may add bots or rename them, so check the list periodically.

CDN and firewall note

robots.txt is a permission request; whether a bot actually reaches your site is often decided by a CDN, WAF or bot-protection layer. A robots.txt that allows a bot while your firewall blocks the same bot is a common silent failure in visibility projects.

  • Check server or CDN logs to confirm that requests from OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot and Googlebot return successful (2xx) responses.
  • OpenAI advises allowing the IP ranges it publishes for OAI-SearchBot, while Anthropic advises against blocking by IP. Read the provider’s own document before writing an IP rule.
  • Bot-protection products may ship one-click "block AI bots" defaults; if your policy is visibility, switch them off or define exceptions.

This checklist is part of our technical crawlability audit; it is a logical consequence of the providers' documents, not a claim of the form "you lose this much".

What robots.txt cannot do

  • It does not guarantee appearance. A bot crawling your site does not mean you will be named in an answer. No provider, OpenAI and Google included, fully publishes its ranking or selection criteria; Perplexity does not document them.
  • It may not stop user-triggered bots. OpenAI documents that robots.txt may not apply to ChatGPT-User, and Perplexity documents that Perplexity-User generally ignores robots.txt.
  • It is not instant. According to OpenAI, changes take about 24 hours to take effect.

Source: OpenAI: Bots ↗; Perplexity: Bots ↗; Anthropic: web crawling ↗; Google: Common crawlers ↗

Can AI bots actually reach your site?

Send us your website and we will review access together, from robots.txt to CDN and log level.

Request an AI Visibility Audit akin@ecommerce.com