Services
Technical AI Crawlability
We audit and fix the technical ground that lets answer engines find, read and, where appropriate, quote your pages.
What is it?
Every engine runs its own bots and lets you manage them separately in robots.txt. If a bot is blocked, it becomes hard to appear in that engine’s answers however good the content is.
This service sets a bot-by-bot access policy for your site and turns it into workable fixes: robots.txt, CDN and WAF rules, server-rendered HTML, sitemaps, indexing and verification tools.
For Google AI Overviews and AI Mode, the condition for appearing is that the page is indexed and eligible to show with a snippet. Google says there are no additional technical requirements and no special “AI file”.
Who delivers
The AI Visibility Turkey team runs the audit; the site-side technical changes are handled by the group’s execution arm Optimum7 (our ecommerce development and digital marketing agency).
What we do
Bot-by-bot robots.txt policy
We tie what is open and what is closed for OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, Googlebot, Bingbot and Google-Extended to a written policy.
CDN and WAF rules
We check that your security layer is not blocking search bots by mistake. OpenAI publishes its allowed IP ranges; Anthropic advises against blocking by IP.
Server-rendered HTML
We verify that key content is in the first HTML response. Key facts should not live only in tabs, accordions, PDFs or images.
Sitemaps and indexing
We check the sitemap, canonical addresses and the index status in Google and Bing.
Verification tools
We set up or check Search Console and Bing Webmaster Tools verification so you can monitor AI performance.
llms.txt assessment
We treat it as an optional, low-cost hygiene step. We do not present it as a ranking lever.
Deliverables
Bot access matrix
For each bot: the current state, the recommended policy and the reasoning.
Prioritized fix list
robots.txt, CDN/WAF and HTML issues in order of impact and effort.
Recommended robots.txt text
A ready-to-apply version written to your policy.
Verification summary
The outcome of the index and access checks after the fixes.
Options and trade-offs
Appear or train?
Allowing search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) lets your site appear in those engines’ answers. Training bots (GPTBot, ClaudeBot) are a separate setting. You can allow one and block the other; it is a business decision, not a technical necessity.
Allow all or allow selectively
Opening everything gives the widest reach; a selective policy lets you limit use of your content for training. A selective policy needs its rules kept up to date.
IP lists or user-agent rules
IP allowlists give tighter control but need upkeep. Because Anthropic advises against blocking by IP, user-agent rules suit those bots better.
User-triggered bots
Bots that act on a user’s request, such as ChatGPT-User and Perplexity-User, may not follow robots.txt. For these bots robots.txt alone is not a firm control.
Decision note: appear or train?
You decide in which engines you want to appear in answers and in which you allow your content to be used for training. We write out the technical consequence of each option plainly.
Evidence status
Documented by the engine
According to OpenAI’s documentation, OAI-SearchBot surfaces sites in ChatGPT search; GPTBot may be used for training and is a separate setting; ChatGPT-User is user-initiated and robots.txt may not apply. Changes take about 24 hours to take effect.
According to Perplexity’s documentation, PerplexityBot surfaces and links sites in Perplexity search and is not used for foundation-model training; Perplexity-User fetches on a user’s request and generally ignores robots.txt. Perplexity does not publish how sources are ranked.
According to Anthropic, ClaudeBot is for training, Claude-User is a user-triggered fetch and Claude-SearchBot indexes for search. Blocking the last two may reduce visibility in Claude search answers.
Google-Extended is a robots.txt token that controls Gemini training and grounding; it does not affect inclusion in Search. Microsoft Advertising (8 October 2025) advises not hiding key answers in tabs or accordions and not leaving information only in PDFs or images.
llms.txt is an informal proposal first published by Jeremy Howard on 3 September 2024; it is not a ratified standard. Reports say Google’s John Mueller has said no AI system uses it and Google Search does not need it. So we treat it as an optional hygiene step.
Source: OpenAI: bots ↗; Perplexity: bots ↗; Anthropic: crawling ↗; Google: common crawlers ↗; Microsoft Advertising: content for AI answers ↗; llmstxt.org ↗; Search Engine Roundtable: llms.txt ↗
Who it is for
It may suit you
- Sites that run behind a security layer (CDN/WAF).
- Sites that build most of their content in the browser (client side).
- Sites that may have blocked all bots with one general rule.
- Organizations that do not know which bots they allow.
Worth knowing
- If your robots.txt is open and your content arrives server-rendered, a short check is often enough; you may not need a larger project.
Process
Read the current state
We review robots.txt, response headers, CDN/WAF logs and index status.
Bot-by-bot policy
We turn your preferences on appearing and training into a written policy for each bot.
Apply the fixes
With your approval we apply the robots.txt, CDN/WAF and HTML fixes.
Verify
We confirm the result through the index, Search Console and Bing Webmaster Tools. Changes on the OpenAI side can take about 24 hours to show.
Pricing
Technical work runs inside the Monthly Program package, or is scoped separately after a Growth Audit; the Monthly Program indicative price range applies, see Packages and Pricing.
What we do not promise
Opening access to bots does not guarantee you appear in answers. For Google’s AI features too, indexing and display are never guaranteed. Access is a necessary condition, not a sufficient one.
FAQ
Should I allow AI bots completely?
If I block Google-Extended, will I drop out of Google Search?
Should I add llms.txt?
Where does your brand stand in AI answers?
Send us your website and topics and we will reply by email.