AI crawler access checklist

AI companies run separate crawlers for search answers, for model training, and for fetching a page someone asked about. You can allow one and block another. Work through these steps in order.

Create a free account

robots.txt

Cited, not trained

Search and answer bots stay allowed. Each training crawler and control token gets its own group with Disallow: /. Add these groups to your existing robots.txt rather than replacing it, and delete any group for a crawler you're happy to let in.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

1. Check what robots.txt says today

Run your homepage through the robots.txt tester at indexzero.site/tools/robots-txt-tester. It fetches the live file, lists its errors, and shows whether each crawler below may fetch the page, with the line that decides. Note the result before you change anything.

2. Decide per job, not per company

Most AI companies publish separate tokens for separate jobs, and blocking one leaves the others alone. Decide for each group:

  • Search engines: Crawl pages so they can appear in a search engine's results. Tokens: Googlebot, Applebot, DuckDuckBot, Amzn-SearchBot.
  • AI search and answer bots: Fetch pages so an AI search product can show or cite them. Tokens: OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot.
  • AI training and dataset crawlers: Collect content that may be used to train models, or control whether it can be. Tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot.
  • User-triggered fetchers: Visit a page because a person asked an assistant to. Several vendors say robots.txt may not apply. Tokens: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User.
  • Previews and ad checks: Fetch a page for a link preview or an ad review. Tokens: facebookexternalhit, OAI-AdsBot.

3. Keep search and answer bots allowed if you want to be cited

These bots are how an AI product shows and links to your pages. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results.

Check that no group names them with Disallow: / by accident, and that the * group they fall back to doesn't block the pages you want cited.

4. Block training in separate groups

A crawler follows only the group that names it, or the * group if none does. A group for GPTBot changes nothing for OAI-SearchBot, so blocking training doesn't cost you ChatGPT search. The robots.txt rules on this page do this for every training crawler and control token the tester knows.

Google-Extended and Applebot-Extended aren't crawlers. Google says Google-Extended does not affect inclusion or ranking in Google Search, and Apple says pages that disallow Applebot-Extended can still appear in its search results.

5. Know which fetchers robots.txt won't stop

ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User fetch a page because a person asked an assistant about it, and their vendors say robots.txt may not apply. If a page must stay private, put it behind a login.

6. Check the rules in front of robots.txt

A CDN, firewall or bot-management rule can block a crawler that robots.txt allows. If you use one, compare its bot settings with the decisions above. The tester only reads robots.txt, and its request is sent as IndexZero-Audit/1.0, so it can't show what a firewall does to GPTBot.

7. Don't use robots.txt to keep pages out of search

Google says a disallowed page can still be indexed if other sites link to it. To keep a page out of results, use a noindex robots meta tag or an X-Robots-Tag header and leave the page crawlable, so Google can see the rule.

8. Wait for caches, then test again

Google usually caches robots.txt for up to 24 hours, and DuckDuckGo says a change reaches DuckAssistBot after 72 hours. Run the tester again once the change is live. For important pages, URL Inspection in your own Search Console shows when Google last crawled them.

Beyond robots.txt

Audit the pages robots.txt lets in

A site audit in IndexZero starts from your sitemap, follows robots.txt, and checks every page it reaches for 27 technical and on-page issue types. The free plan audits up to 50 pages per run, paid from its 2,000 monthly credits. No card needed.

Create a free accountTest robots.txt now

More resources