OpenAI's crawlers
| Crawler | What it does | Allow it? |
|---|---|---|
| OAI-SearchBot | Finds pages to show in ChatGPT search answers | Yes, if you want AI visibility |
| GPTBot | Collects content that may train OpenAI's models | Your choice; separate from search |
| ChatGPT-User | Visits pages when a user asks ChatGPT to | Not used for search eligibility |
| OAI-AdsBot | Checks landing pages submitted as ads | Needed if you advertise |
OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", and that a robots.txt change takes about 24 hours to take effect.
A simple robots.txt for B2B
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you want to block training but keep search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Don't forget the host
OpenAI notes that your host or CDN must also allow traffic from its published crawler IP addresses. Some security settings block crawlers by default; check them.
How to check what's allowed today
- Open yourdomain.com/robots.txt.
- Look for rules naming OAI-SearchBot, GPTBot or ChatGPT-User.
- Look for a blanket "Disallow: /" under "User-agent: *".
- Check your CDN or security plugin for bot-blocking settings.
Questions
Will allowing GPTBot help us get recommended?
Training data shapes answers given without search. Whether to allow it is a business choice; it's separate from appearing in ChatGPT search.
Do other AI assistants use their own crawlers?
Yes, several do. Check each one's documentation if you want to manage them individually; most B2B sites simply allow them.
Can crawlers slow down our site?
Rarely at B2B traffic levels. If they do, rate-limit rather than block.
Common reasons sites block AI crawlers by accident
- A security plugin or CDN rule blocking "bots" broadly.
- A robots.txt copied from a template that disallows unknown agents.
- Staging rules left in place after launch.
- Rate limits set so low that crawlers give up.
Any of these can keep you out of ChatGPT search answers without anyone noticing.
A quick test
Check your server logs or CDN analytics for OAI-SearchBot visits after allowing it. OpenAI says changes take about 24 hours to take effect; if the crawler still doesn't appear after a few days, look for blocks at the host level.
Start here: AI visibility for B2B. More articles in AI visibility.
