---
title: "Check that your CDN and firewall allow search crawlers"
description: "Allowing crawlers in robots.txt isn't enough if your CDN, firewall or bot protection blocks them first."
url: https://aeo.so/tactics/check-that-your-cdn-and-firewall-allow-search-crawlers
author: "Paul Andre de Vera"
updated: 2026-11-15
---

# Check that your CDN and firewall allow search crawlers

Allowing crawlers in robots.txt isn't enough if your CDN, firewall or bot protection blocks them first. Google's guidance for AI features says to confirm your hosting and CDN allow crawling. Check bot settings for Googlebot, Bingbot and AI search crawlers, and test key pages with URL Inspection and your server logs.

- Area: [Technical Foundation](https://aeo.so/categories/technical-foundation)
- Credited to: [Google Search Central](https://aeo.so/experts/google-search-central)
- Part of: [the aeo.so answer engine optimization checklist](https://aeo.so/categories)

The worked example, why this helps with AI search, and the original source are free for members: https://aeo.so/sign-in


Read the guide:

- [What is answer engine optimization (AEO)?](https://aeo.so/what-is-answer-engine-optimization)
- [AEO vs SEO: what changes and what stays the same](https://aeo.so/guides/aeo-vs-seo)
- [What is AI SEO?](https://aeo.so/guides/ai-seo)

## More in Technical Foundation

- [Use snippet controls to manage what AI features show](https://aeo.so/tactics/use-snippet-controls-to-manage-what-ai-features-show)
- [Put key details in HTML text, not PDFs or images](https://aeo.so/tactics/put-key-details-in-html-text-not-pdfs-or-images)
- [Generate API documentation for AI consumption](https://aeo.so/tactics/generate-api-documentation-for-ai-consumption)
- [Add AI-specific user agents to robots.txt](https://aeo.so/tactics/add-ai-specific-user-agents-to-robots-txt)
- [Create an XML sitemap and submit to search engines](https://aeo.so/tactics/create-an-xml-sitemap-and-submit-to-search-engines)
