Which AI crawlers should you know?

CompanyUser agentPurposeFollows robots.txt?
OpenAIOAI-SearchBotIndex for ChatGPT search answersYes
OpenAIGPTBotModel trainingYes
OpenAIChatGPT-UserFetches a page a user asks forMight not apply
AnthropicClaude-SearchBotIndex for Claude's search resultsYes
AnthropicClaudeBotModel trainingYes
AnthropicClaude-UserFetches a page a user asks forYes
PerplexityPerplexityBotIndex for Perplexity answers; not used for trainingYes
PerplexityPerplexity-UserFetches a page a user's question needsGenerally no
GoogleGooglebotGoogle Search, including AI Overviews and AI ModeYes
GoogleGoogle-ExtendedA token controlling Gemini training use; not a separate crawlerYes
MicrosoftBingbotBing Search and Copilot answersYes

The bold user agents decide if you can be cited in AI search. The others govern training and live fetches.

What is the difference between training and search crawlers?

Training crawlers collect text that can be used to build future AI models. Blocking them affects what future models learn, not whether today's AI search can cite you. Search crawlers build the index that AI answers retrieve from. Blocking them removes you from that tool's citations.

OpenAI says it plainly: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic warns that blocking Claude-SearchBot or Claude-User can reduce your visibility in Claude's search results.

Name each crawler you want to allow, keep private areas out, and point to your sitemap. This is aeo.so's own robots.txt. The /md/ rule hides an internal path; the public Markdown pages stay open.

User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: Bingbot
User-Agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/

User-Agent: *
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/

Host: https://aeo.so
Sitemap: https://aeo.so/sitemap.xml

To keep content out of training but stay citable, add a Disallow: / group for GPTBot, ClaudeBot and Google-Extended, and keep the search crawlers allowed.

What else can block AI crawlers?

robots.txt is only one gate. Check three more:

  1. Your CDN or firewall. Bot protection can block AI crawlers before robots.txt is read. OpenAI and Perplexity publish IP ranges you can allow.
  2. JavaScript-only content. Many AI crawlers do not run scripts. Render content on the server.
  3. Login walls and interstitials. Crawlers cannot get past them.

Check your server logs for each user agent to confirm they reach your pages.

How fast do changes take effect?

OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search. Other crawlers reread robots.txt on their own schedules, often within a day. Google rereads it regularly too. After any change, check your logs and test a question in each tool a few days later.