1. Home (EN)
  2. Blog (EN)
  3. Robots.txt for AI Crawlers: The Complete Configuration Guide

GEO Blog

Robots.txt for AI Crawlers: The Complete Configuration Guide

By José Felipe Bello, Founder of Bello GEO · June 2026 · 5 min read · Versión en Español
“Blocking GPTBot or ClaudeBot in robots.txt is asking the AI to recommend you without letting it read your site. AI-crawler access weighs 10% of the Bello GEO Visibility Index.” — José Felipe Bello, founder of Bello GEO

Your robots.txt file is the first door AI crawlers find when they visit your site. Block them — intentionally or by accident — and you are invisible to ChatGPT, Claude, Perplexity and Google AI. Allow them correctly and you give them access to the content they need in order to cite you.

In our audits, more than 60% of the Colombian websites we have reviewed block at least one AI crawler without knowing it — which means they are invisible to AI answers without ever having decided to be.

The AI crawlers you need to know

  • GPTBot — OpenAI's crawler. Block it and ChatGPT cannot access your content to cite you.
  • ChatGPT-User — fires when a ChatGPT user triggers web search; separate from GPTBot.
  • ClaudeBot — Anthropic's crawler, indexing content for Claude's answers.
  • PerplexityBot — indexes aggressively for Perplexity.ai answers.
  • Google-Extended — controls Gemini's access, separate from Googlebot.
  • Googlebot — still the most important: it feeds both Google Search and Google AI Overviews.

Recommended configuration

For maximum AI visibility, your robots.txt should explicitly allow all AI crawlers with Allow directives, and reference both your sitemap and your llms.txt. Explicit allows also protect you when a future default changes under your feet.

Common mistakes

  • Global Disallow: a "Disallow: /" under User-agent: * blocks every crawler — AI included.
  • WordPress defaults: some themes block resources crawlers need beyond /wp-admin/.
  • Firewall/CDN blocking: some firewalls and CDNs block AI crawlers at the network level, before your robots.txt is ever read — robots.txt looks fine, access is still denied. Test with real fetches, not just file review.

How to verify

Checking real AI-crawler access — robots.txt, network level and rendering — is one of the first steps of every Bello GEO audit: the partial Visibility Index diagnosis is free, and the complete audit (USD $1,500, included at no extra cost in a full implementation) covers crawler access as one of its 8 dimensions.

José Felipe Bello is the founder of Bello GEO, the first bilingual GEO agency specialized in Latin America. He is also co-founder and CTO of Laboratorio del Dolor, the clinic that serves as the agency's founding case (Visibility Index 51 → 89 in 6 weeks) — a relationship we disclose in every mention of the case.

How visible is your business to AI?

Get your free partial Visibility Index diagnosis. The complete audit is USD $1,500 — included at no extra cost in a full implementation.

Start on WhatsApp