Which AI crawlers should you know?
| Company | User agent | Purpose | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Index for ChatGPT search answers | Yes |
| OpenAI | GPTBot | Model training | Yes |
| OpenAI | ChatGPT-User | Fetches a page a user asks for | Might not apply |
| Anthropic | Claude-SearchBot | Index for Claude's search results | Yes |
| Anthropic | ClaudeBot | Model training | Yes |
| Anthropic | Claude-User | Fetches a page a user asks for | Yes |
| Perplexity | PerplexityBot | Index for Perplexity answers; not used for training | Yes |
| Perplexity | Perplexity-User | Fetches a page a user's question needs | Generally no |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | A token controlling Gemini training use; not a separate crawler | Yes | |
| Microsoft | Bingbot | Bing Search and Copilot answers | Yes |
The bold user agents decide if you can be cited in AI search. The others govern training and live fetches.
What is the difference between training and search crawlers?
Training crawlers collect text that can be used to build future AI models. Blocking them affects what future models learn, not whether today's AI search can cite you. Search crawlers build the index that AI answers retrieve from. Blocking them removes you from that tool's citations.
OpenAI says it plainly: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic warns that blocking Claude-SearchBot or Claude-User can reduce your visibility in Claude's search results.
How should you set up robots.txt for AI search?
Name each crawler you want to allow, keep private areas out, and point to your sitemap. This is aeo.so's own robots.txt. The /md/ rule hides an internal path; the public Markdown pages stay open.
User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: Bingbot
User-Agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/
User-Agent: *
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/
Host: https://aeo.so
Sitemap: https://aeo.so/sitemap.xml
To keep content out of training but stay citable, add a Disallow: / group for GPTBot, ClaudeBot and Google-Extended, and keep the search crawlers allowed.
What else can block AI crawlers?
robots.txt is only one gate. Check three more:
- Your CDN or firewall. Bot protection can block AI crawlers before robots.txt is read. OpenAI and Perplexity publish IP ranges you can allow.
- JavaScript-only content. Many AI crawlers do not run scripts. Render content on the server.
- Login walls and interstitials. Crawlers cannot get past them.
Check your server logs for each user agent to confirm they reach your pages.
How fast do changes take effect?
OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search. Other crawlers reread robots.txt on their own schedules, often within a day. Google rereads it regularly too. After any change, check your logs and test a question in each tool a few days later.