robots.txt tester and AI crawler checker

It fetches /robots.txt from the origin of the URL you enter and tests the path of that URL, or / if you enter a bare domain. The status code, redirects, size and file contents are measured from that live request, sent with the User-Agent IndexZero-Audit/1.0. A server that treats bots differently can send Googlebot or GPTBot a different response.

Allowed or blocked is worked out from the file, the same way Google's parser matches rules. It shows what the file asks for. Crawlers that don't follow robots.txt can still fetch the page, and the table says which ones their vendors describe that way.

You can change the path or type any user agent token after the check. The file is already loaded, so the result updates without fetching again.

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. Google reads four fields: user-agent, allow, disallow and sitemap. It ignores the rest, including crawl-delay.

Which group applies

A crawler uses the group with the most specific user agent that matches its own token. If no group names it, it uses the * group. If a token appears in several groups, Google combines their rules into one group. A crawler follows only its own group, so a GPTBot group with a single rule replaces everything under * for GPTBot.

This tester matches each crawler's token exactly, ignoring case and any version number. One vendor documents a fallback of its own: Applebot follows the Googlebot group when robots.txt doesn't name Applebot, and the table applies it.

Which rule wins

• Rules match from the start of the path, and paths are case-sensitive: Disallow: /Admin does not block /admin.
• When several rules match, the one with the longest path wins.
• When an Allow and a Disallow rule are the same length, Google uses the less restrictive rule, so Allow wins.
• * matches any sequence of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?download=1.
• Rules placed before the first User-agent line belong to no group, and Google ignores them.

What the status code means

• 2xx: crawlers read the file as served.
• 4xx other than 429, including 404: Google treats this as no robots.txt, so there are no restrictions.
• 5xx, 429, or a DNS or network failure: Google treats this as a server error. For the first 12 hours it stops crawling the whole site while it retries. For the next 30 days it uses the last copy it fetched successfully.
• Redirects: Google follows at least five hops, then treats the file as a 404.
• Size: Google reads the first 500 KiB of the file and ignores the rest.

Google usually caches robots.txt for up to 24 hours, so a fix can take a day to reach Googlebot.

Google describes robots.txt as a way to control crawling, and says it is not a mechanism for keeping a web page out of Google. A page that's disallowed can still be indexed if other sites link to it. Google then shows the URL, and possibly the link's anchor text, without a description.

To keep a page out of results, use a noindex robots meta tag or an X-Robots-Tag: noindex response header, or put the page behind a password. Leave the page crawlable. If robots.txt blocks it, Google never sees the noindex rule and can keep the page in results.

A noindex line inside robots.txt does nothing. Google retired support for noindex, nofollow and other unpublished robots.txt rules on September 1, 2019. The tester marks that line as an error.

The AI companies publish separate tokens for different jobs, so you don't have to choose between being cited and being used for training. Check each token against what its vendor says it does.

Search and answer bots

These let an AI product show and link to your pages. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers. Perplexity says PerplexityBot surfaces and links sites in its results and is not used to crawl content for AI foundation models. Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results. DuckDuckGo says DuckAssistBot feeds its cited AI answers and is not used for training.

Training crawlers and control tokens

GPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta) and Amazonbot (Amazon) collect content their vendors say may be used to train models. Google-Extended and Applebot-Extended are not crawlers. They are tokens Google and Apple read to decide whether content their existing crawlers fetched may be used for Gemini and Apple's foundation models. Google says Google-Extended does not affect inclusion or ranking in Google Search. Apple says pages that disallow Applebot-Extended can still appear in its search results.

If you want to be cited but not used for training, give each training token its own group with Disallow: / and leave the search bots allowed. A group for GPTBot does not change what OAI-SearchBot may fetch, because each crawler follows only its own group, or * if it has none.

User-triggered fetchers

When someone asks an assistant to read a page, the fetch uses a separate token, and several vendors say robots.txt may not stop it. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores them. Meta says meta-externalfetcher may bypass them, and Amazon says Amzn-User may not follow every directive. Anthropic says Claude-User respects robots.txt. If content must not be fetched at all, protect it with a login.

GET /api/tools/robots-txt?url=https://example.com/blog returns the same result as JSON. Add path=/some/path to test a different path, and ua=MyCrawler to add one user agent token to the result. The response includes the raw file (up to 500 KiB), the status interpretation, the findings with line numbers, the sitemaps, and allowed or blocked for every crawler in the table with the line that decided.