---
title: "Add AI-specific user agents to robots.txt"
description: "Write explicit robots.txt allow rules for GPTBot, ClaudeBot, PerplexityBot and Google-Extended. AI systems can only cite pages their crawlers can read."
url: https://aeo.so/tactics/add-ai-specific-user-agents-to-robots-txt
author: "Paul Andre de Vera"
updated: 2026-11-15
---

# Add AI-specific user agents to robots.txt

Adding AI-specific user agents to robots.txt means writing explicit allow rules for crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. AI systems can only use pages their crawlers are permitted to read, and clear rules make your content available for AI answers and citations.

- Area: [Technical Foundation](https://aeo.so/categories/technical-foundation)
- Credited to: [Guerin Green](https://aeo.so/experts/guerin-green)
- Part of: [the aeo.so answer engine optimization checklist](https://aeo.so/categories)

The worked example, why this helps with AI search, and the original source are free for members: https://aeo.so/sign-in


Read the guide:

- [What is answer engine optimization (AEO)?](https://aeo.so/what-is-answer-engine-optimization)
- [What is AI SEO?](https://aeo.so/guides/ai-seo)
- [How ChatGPT chooses sources, and how to get cited](https://aeo.so/guides/chatgpt-citations)

## More in Technical Foundation

- [Create an XML sitemap and submit to search engines](https://aeo.so/tactics/create-an-xml-sitemap-and-submit-to-search-engines)
- [Implement proper heading structure (H1-H6)](https://aeo.so/tactics/implement-proper-heading-structure-h1-h6)
- [Optimize images with descriptive alt text](https://aeo.so/tactics/optimize-images-with-descriptive-alt-text)
- [Set up Google Search Console and Bing Webmaster Tools](https://aeo.so/tactics/set-up-google-search-console-and-bing-webmaster-tools)
- [Use HTTPS encryption for all pages](https://aeo.so/tactics/use-https-encryption-for-all-pages)
