AI crawler access
AI crawler access: which bots can read your site
Check named AI search crawlers against robots.txt. Uses no credits. Training-bot blocks are a preference, not a ranking defect. Hosted billing is paused; this check is not a $10 SKU.
What you can do
- 01
Check robots.txt
SearchCrew fetches public robots.txt and maps named crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot.
- 02
Separate search from training
A GPTBot block does not by itself remove you from ChatGPT Search. Search crawlers and training crawlers are different rules.
- 03
Run it in the app or through MCP
The same credit-free check is available in a SearchCrew project and as analyze_ai_crawler_access for Squadbots and other MCP clients.
Data you can act on
- Search crawlers
- Access map
- Training crawlers
- Preference
- Credits
- None
- SKU
- None
Use cases
- See whether ChatGPT Search, Claude, Perplexity, or Googlebot is blocked before you rewrite pages for AI answers.
- Confirm a training-bot block is intentional instead of treating it as a defect.
Why SearchCrew
- No credits for this check. Pay is paused and there is no hosted $10 SKU.
- This report is an access map, not a ranking or citation prediction.
Related features
FAQ
Does this use SearchCrew credits?
No. The crawler-access check reads public robots.txt and does not spend DataForSEO credits.
If I block GPTBot, am I hidden from ChatGPT Search?
No. GPTBot is a training crawler. ChatGPT Search uses OAI-SearchBot. Allow or block them separately.
Does missing robots.txt block AI crawlers?
No. Missing robots.txt, or an empty Disallow, is treated as allow-all for the named crawlers in this report.
What does this cost?
Hosted billing is paused. This check is not sold as a $10 SKU. Self-host if you need it today.
Check launch status
The focused alternative to bloated, expensive, legacy SEO suites.
SearchCrew