---
title: "AI crawlers explained: GPTBot, OAI-SearchBot, ClaudeBot and more"
description: "Every major AI crawler in one table: which ones train models, which power AI search, which follow robots.txt, and how to set your robots.txt."
url: https://aeo.so/guides/ai-crawlers
author: "Paul Andre de Vera"
updated: 2026-11-15
---

# AI crawlers explained: GPTBot, OAI-SearchBot, ClaudeBot and more

By Paul Andre de Vera. Updated 2026-11-15.

AI crawlers are the bots AI companies use to read websites. Most companies now run separate bots for training models, for indexing pages for AI search, and for fetching a page when a user asks. For AEO, allow the search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and decide separately whether to allow the training crawlers.

## Which AI crawlers should you know?

| Company | User agent | Purpose | Follows robots.txt? |
|---|---|---|---|
| OpenAI | **OAI-SearchBot** | Index for ChatGPT search answers | Yes |
| OpenAI | GPTBot | Model training | Yes |
| OpenAI | ChatGPT-User | Fetches a page a user asks for | Might not apply |
| Anthropic | **Claude-SearchBot** | Index for Claude's search results | Yes |
| Anthropic | ClaudeBot | Model training | Yes |
| Anthropic | Claude-User | Fetches a page a user asks for | Yes |
| Perplexity | **PerplexityBot** | Index for Perplexity answers; not used for training | Yes |
| Perplexity | Perplexity-User | Fetches a page a user's question needs | Generally no |
| Google | **Googlebot** | Google Search, including AI Overviews and AI Mode | Yes |
| Google | Google-Extended | A token controlling Gemini training use; not a separate crawler | Yes |
| Microsoft | **Bingbot** | Bing Search and Copilot answers | Yes |

The bold user agents decide if you can be cited in AI search. The others govern training and live fetches.

## What is the difference between training and search crawlers?

Training crawlers collect text that can be used to build future AI models. Blocking them affects what future models learn, not whether today's AI search can cite you. Search crawlers build the index that AI answers retrieve from. Blocking them removes you from that tool's citations.

OpenAI says it plainly: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic warns that blocking Claude-SearchBot or Claude-User can reduce your visibility in Claude's search results.

## How should you set up robots.txt for AI search?

Name each crawler you want to allow, keep private areas out, and point to your sitemap. This is aeo.so's own [robots.txt](https://aeo.so/robots.txt). The `/md/` rule hides an internal path; the public Markdown pages stay open.

```text
User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: Bingbot
User-Agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/

User-Agent: *
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/
Disallow: /auth/
Disallow: /sign-out
Disallow: /sign-in/stop
Disallow: /verify-email
Disallow: /md/

Host: https://aeo.so
Sitemap: https://aeo.so/sitemap.xml
```

To keep content out of training but stay citable, add a `Disallow: /` group for GPTBot, ClaudeBot and Google-Extended, and keep the search crawlers allowed.

## What else can block AI crawlers?

robots.txt is only one gate. Check three more:

1. **Your CDN or firewall.** Bot protection can block AI crawlers before robots.txt is read. OpenAI and Perplexity publish IP ranges you can allow.
2. **JavaScript-only content.** Many AI crawlers do not run scripts. Render content on the server.
3. **Login walls and interstitials.** Crawlers cannot get past them.

Check your server logs for each user agent to confirm they reach your pages.

## How fast do changes take effect?

OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search. Other crawlers reread robots.txt on their own schedules, often within a day. Google rereads it regularly too. After any change, check your logs and test a question in each tool a few days later.

## Frequently asked questions

### Should I block GPTBot?

Block GPTBot if you do not want OpenAI to train models on your content; allow it if you are happy for it to. Either way, blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot. Most sites focused on AEO allow the search crawlers and make a separate choice about training.

### Do AI crawlers run JavaScript?

Many AI crawlers read the HTML your server sends and do not run JavaScript the way a browser does. Content that only appears after scripts run can be invisible to them. Server-side rendering or static pages, as aeo.so uses, keep your content readable by every crawler.

### Are Claude-Web and anthropic-ai still valid robots.txt tokens?

No. Anthropic's current crawlers are ClaudeBot, Claude-SearchBot and Claude-User. The older tokens Claude-Web and anthropic-ai no longer match its bots, and rules naming them have no effect. Update old block lists to the current names, and check them against each company's own documentation.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
