Short answer
OpenAI needs OAI-SearchBot to fetch product image URLs from your feed. If your image host, CDN or firewall returns a 403, feed processing can fail. Allowlist OpenAI's crawlers by user agent, check image hosts as well as your main domain, and avoid challenges that block bots.
Fix
- Allow OAI-SearchBot and OAI-AdsBot in robots.txt
- Allowlist them in Cloudflare, Akamai or your WAF (OAI-AdsBot is verified by Cloudflare)
- Check the image CDN separately
- Use openai.com/searchbot.json and adsbot.json if you need IP lists
- Re-upload or resubmit affected items if status doesn't update
Example robots.txt
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /Questions
Should we also allow GPTBot?
GPTBot is for model training and is a separate decision. It isn't needed for ads or product feeds.
Facts checked against the sources below on 12 October 2026. Platforms change quickly: if something here no longer matches what you see, tell us and we will update it.
