Short answer
Bot protection in Cloudflare, Akamai or a WAF often returns 403 errors to AI crawlers, so pages never reach AI search or ad review. Allowlist the crawlers you want by verified user agent, exempt them from CAPTCHAs and JavaScript challenges, and check logs for 403 and 429 responses.
Checks
- Search logs for OAI-SearchBot, OAI-AdsBot, Googlebot, Bingbot
- Look for 403, 429 or challenge pages
- Use verified-bot features (OAI-AdsBot is verified in Cloudflare)
- Combine user agent with published IP lists
- Retest after changes
Questions
Isn't allowing bots risky?
Allow specific verified crawlers, not all bots.
Facts checked against the sources below on 12 October 2026. Platforms change quickly: if something here no longer matches what you see, tell us and we will update it.
