Cloudflare Settings for SEO & GEO and AI Search Bots

Cloudflare security: stopping harmful bots without closing AI Search access

When ChatGPT, Perplexity, or another crawler gets a 403 Forbidden response or a Just a moment... page, the failure can happen before the content is read. That does not prove the content is poor, and it does not mean every protection should be disabled. Read Security Events, identify the product and rule that acted, then narrow the policy using a bot identity Cloudflare actually verifies.

Cloudflare separates several layers: the Web Application Firewall (WAF), custom rules, Bot Fight Mode, Super Bot Fight Mode, rate limiting, and AI crawler controls. Their evaluation order and interoperability limits differ. A Skip rule therefore does not automatically bypass every product. In particular, a custom WAF rule cannot bypass Bot Fight Mode, so do not publish an allowlist that looks correct but cannot reach the crawler.

1. The Bot Fight Mode and Block AI Bots trap

Bot Fight Mode is useful against simple automation, but it is not a precise crawler-permission system. If Security Events show that it challenges a crawler you intentionally want to assess, a custom rule cannot bypass it. Either disable that mode after a risk review, use a more granular bot control where your plan supports it, or narrow protection to the paths that truly need it.

Likewise, Block AI Bots or AI Crawl Control can block AI crawler categories. That can fit a content-use policy, but it is not an SEO button. When the block includes a crawler you expect to discover your public pages, robots.txt, sitemap.xml, and llms.txt still cannot be fetched. Part 1 covers those documents; this article makes sure they do not stop at the firewall.

Check this before changing a toggle

• Test the same public URL that failed, not only the homepage.
• Open Security Events and find Block, Managed Challenge, or JS Challenge actions at the test time.
• Record the rule or product, URI, and bot status; never open the whole site because of one user-agent string.
• Keep /admin/, private APIs, login, checkout, and uploads protected even when public articles remain crawlable.

2. Create a Custom WAF Rule for genuinely verified search bots

Open Security > WAF > Custom rules, but do not create a global allowlist first. Start with logging or simulation where available, then inspect the events. When the goal is to keep verified crawlers clear of another WAF control, use a Cloudflare bot signal and choose Skip only for the relevant components. Do not use a broad Allow action as a shortcut because it can suppress protections that should still run.

# Use only where the zone exposes a known/verified-bot signal.

# Example: public /robots.txt only; confirm the need in Security Events.

# Action: Skip → select only relevant custom or managed WAF rules.

(cf.client.bot and http.request.uri.path eq "/robots.txt")

This signal includes verified bots beyond search crawlers. Never apply Skip globally: limit it to public paths that actually need an exception, and do not expose login, dashboard, or private API paths. Enterprise Bot Management fields may depend on the account plan. Test field availability in the expression builder before use; do not force a rule that fails validation merely because it appears in an online example.

# Enterprise only: use only if the expression builder accepts this field.

(cf.bot_management.verified_bot and http.request.uri.path eq "/robots.txt")

Bot names remain useful for observation, not as security identity. The expression below is appropriate for a temporary Log rule while investigating “ChatGPT cannot access a Cloudflare site.” It is not a production allowlist: an attacker can send the same headers.

(http.user_agent contains "OAI-SearchBot" or

 http.user_agent contains "PerplexityBot" or

 http.user_agent contains "Claude-SearchBot" or

 http.user_agent contains "Googlebot" or

 http.user_agent contains "bingbot")

ChatGPT-User should not become an automatic pass around authentication or rate limits either. A user-triggered request differs from an indexing crawler. Protect private data in the application rather than trusting robots or a user agent as access control. After changing a rule, repeat the event check and confirm the public page now returns its normal response.

3. Read Cloudflare AI Audit and AI Crawl Control as evidence

Cloudflare dashboard labels can vary by plan and product updates. Look for AI Audit or AI Crawl Control to manage AI crawler policy, then use Security Analytics and Security Events as the request evidence. The goal is not to chase a crawl count. It is to see who requested a page, whether the request was allowed or challenged, and which paths are being targeted.

Keep content policy and security policy separate. If you want to limit training crawlers but assess search crawlers, create an explicit policy per category or identity and document why. If traffic is an unverified scraper, retain rate limits, WAF controls, and a sensible cache strategy. AI Audit does not guarantee that an AI system will cite a page; it helps observe and control requests passing through Cloudflare.

4. DNS, speed, and the limits of the AICLUB Cloudflare infrastructure

For an AICLUB subdomain pointed by CNAME to GitHub Pages, Vercel, Netlify, or Cloudflare Pages, the AICLUB record is DNS-Only (grey cloud). This lets the destination host complete its own custom-domain and TLS setup without a double proxy. DNS-Only is not automatically faster or slower: HTTP requests do not pass through AICLUB's Cloudflare proxy, so HTTP/3, caching, Brotli/Zstd compression, WAF, and Early Hints are controlled by the destination host or CDN.

If you operate your own proxied Cloudflare zone, enable speed features only after measuring their effect. HTTP/3 (QUIC) can help supported clients, but it does not replace lean HTML, compressed images, origin caching, and healthy TTFB. Early Hints (103) can let a browser start fetching selected resources earlier, but wrong hints waste bandwidth. 0-RTT has replay considerations, so do not enable it for non-idempotent application endpoints without understanding the app risk.

A too many redirects problem during a CNAME setup is not a reason to switch proxy status blindly. Check the canonical host at the destination, HTTP-to-HTTPS redirects, the custom-domain configuration, and whether two CDNs are trying to manage TLS. For SEO and GEO the target is simple: canonical URLs respond consistently, essential resources are not blocked, and public pages are fast for people and legitimate crawlers.

Need a clean address for a static-hosted project? Browse the AICLUB domain catalog. After launch, use the AICLUB Badge Builder for an additional slot when eligible, then discuss your configuration in the AICLUB Telegram community.

FAQ: Cloudflare settings for SEO and GEO

Technical references

• Cloudflare: Bot Fight Mode
• Cloudflare: stop malicious bots
• Cloudflare: manage AI crawlers
• Cloudflare: Bot Management variables
• Cloudflare: HTTP/3
• Cloudflare: Early Hints

Continue with Part 3: content structure that readers and answer systems can inspect clearly.