Robots Tester

Robots Tester fetches any site’s robots.txt, lints it for mistakes, and lets you test any URL against any crawler (Googlebot, Bingbot, or every major AI bot) showing you the exact line that decided the verdict. Google removed the robots.txt tester from Search Console, so this fills the gap: a real RFC 9309 matcher, a live access matrix for AI crawlers, and plain-English findings for the errors that quietly take pages out of search.

What robots.txt actually does

robots.txt is a plain-text file at the root of your domain (/robots.txt) that tells cooperative crawlers which paths they’re allowed to fetch. It is a directive, not a lock: well-behaved bots honour it, but it doesn’t remove pages from the index and it doesn’t stop bad actors. Two things people constantly conflate:

• Crawling vs indexing. robots.txt controls crawling. To keep a page out of Google’s index you need a noindex meta tag or header, and ironically, if you block the page in robots.txt, Google can’t crawl it to see the noindex, so it may stay indexed with no snippet.
• Blocking assets. Disallowing your CSS or JavaScript directories used to be common advice and is now actively harmful: Google renders pages, and a blocked stylesheet can wreck how your page is understood.

The right instinct is minimalism: leave robots.txt almost empty, and reach for it only for genuinely private or wasteful paths (faceted-search URLs, internal search results, admin areas).

How the rules are resolved (RFC 9309 precedence)

The trap in robots.txt is that “the first matching rule wins” is not how it works. The standard, formalised as RFC 9309, is longest-match wins, with a tie-breaker:

1. Group selection. A crawler picks the group whose User-agent token is the most specific match for its own name. GPTBot obeys a User-agent: GPTBot group over the catch-all User-agent: * group. If there’s no specific group, * applies; if there’s no * group either, everything is allowed.
2. Rule matching. Within that group, every Allow and Disallow pattern is compared to the URL path. * matches any run of characters and $ anchors the end. The longest pattern that matches wins.
3. Tie-break. If an Allow and a Disallow match with the same length, Allow wins.

So Disallow: /downloads/ plus Allow: /downloads/free/ lets /downloads/free/report.pdf through because the Allow pattern is longer. An empty Disallow: means “allow everything.” Robots Tester runs this exact algorithm client-side and scroll-highlights the winning line in the file view, so you never have to reason about precedence in your head.

The AI-crawler landscape

The newest reason people open robots.txt is to decide what AI systems may do with their content, and the crucial point is that “AI crawler” is not one thing. There are three distinct purposes, and blocking them means very different things:

• Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider…) collect content to train future models. Blocking these is a legitimate content-rights choice and costs you nothing in terms of being found today.
• Assistant-fetch bots (ChatGPT-User, Claude-User, Perplexity-User…) fetch a page in real time because a user just asked the assistant about it. Block these and AI tools can’t open or cite you when a customer is actively looking.
• AI-search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot…) build the indexes behind AI answer engines. Block these and you’re invisible in AI search results.

Robots Tester’s access matrix groups every known bot by vendor and tags each with its purpose, so a single glance tells you whether you’re merely opting out of training or accidentally hiding from the entire AI-answer economy. We inform; you decide.

The mistakes that cost the most

Most robots.txt damage comes from a handful of errors, and the tester’s linter calls each one out inline. The catastrophic one is a bare Disallow: / on the * group: that single line makes your whole site invisible to search, and it’s usually left over from a staging config that shipped to production. Others are quieter: rules written before their User-agent line (orphaned and ignored), patterns that aren’t slash-rooted (Disallow: foo instead of Disallow: /foo), a relative Sitemap: URL (it must be absolute), a stray UTF-8 byte-order mark on line one, or a robots.txt that actually returns your styled 404 HTML page, which means you effectively have no robots.txt at all. Each finding comes with the fix, not just the complaint.

For AI agents

This tool exposes a plain-text endpoint designed for language models and agents:

GET https://nitrosites.com/tools/robots-tester/scan.php?url=<target>&format=agent

It returns a token-cheap digest of the same analysis shown on this page. Soft rate limits apply; results are cached briefly. No key or login required.

Frequently asked questions

What does robots.txt actually control?

It tells well-behaved crawlers which paths they may fetch. It does not remove pages from the index (use noindex for that) and it doesn't stop bad actors: it's a directive for cooperative bots like Googlebot, Bingbot and the major AI crawlers.

Which rule wins when several match a URL?

Under RFC 9309 the longest matching path wins, and if an Allow and a Disallow match with equal length, Allow wins. This tester shows you exactly which line decided the verdict.

Should I block AI crawlers like GPTBot or ClaudeBot?

It's a trade-off. Blocking training bots keeps your content out of future model training, but blocking assistant and AI-search bots (like ChatGPT-User or Claude-SearchBot) means AI tools can't cite or recommend your site. The matrix here separates the two so you can choose deliberately.

Why is "Disallow: /" so dangerous?

One character too many and your entire site is invisible to search engines. It's the most expensive robots.txt mistake there is: this tool flags it in red the moment it sees it.

Does robots.txt affect my Google rankings?

Indirectly and powerfully: blocked pages can't be crawled, so they can't rank (and blocked CSS/JS can hurt rendering). Keep robots.txt minimal and use it only for genuinely private or wasteful paths.