π€ Robots.txt Tester
Paste a robots.txt, test URLs, see exactly what each bot can crawl.
Everything runs in your browser. Your content never leaves your device.
How to Use This Tester
Paste the full contents of a robots.txt file on the left, list the URLs you care about, pick the crawler, and get a verdict for every URL: allowed or blocked, which rule decided it, and why that rule won.
Paste your robots.txt. The field starts with a realistic sample you can edit or replace with your own file.
Enter one URL per line. Full URLs (https://example.com/admin) and bare paths (/admin) both work; only the path is tested.
Choose the user-agent. Rules are grouped by user-agent in robots.txt, and a bot follows only its own group, or the * group if it has none.
Read the verdict table. Each row shows Allowed or Blocked, the exact rule that matched, and the reason it won over the other rules.
How Rule Matching Works
Google's matching logic is precise and this tester implements it exactly. First, the crawler finds its group: the group naming its user-agent exactly, or the * group if none names it. Only that group's rules apply; other groups are ignored. Next, every Allow and Disallow pattern is tested against the URL path. A pattern matches when the path starts with the pattern text, where * stands for any characters and a trailing $ requires the URL to end there. Of all matching rules, the longest pattern wins, because a longer pattern is more specific. If an Allow and a Disallow tie at the same length, Allow wins. When nothing matches, crawling is allowed by default. Note that matching is case-sensitive and includes the query string.
Worked Example
Take this robots.txt:
Test /admin/login with the * user-agent. Two rules match: "Disallow: /admin/" (7 characters) and "Allow: /admin/login" (12 characters). The longer Allow wins, so the verdict is Allowed, and the explanation says exactly that. Test /admin/dashboard: only "Disallow: /admin/" matches, so the verdict is Blocked. Test /admin/login with the Googlebot user-agent against the sample file above: Googlebot has its own group, so the * group's admin rules are ignored entirely, no rule in the Googlebot group matches /admin/login, and the verdict is Allowed by default. This group isolation is why testing with the right user-agent matters.
Robots.txt Mistakes That Hurt Sites
The costliest mistake is blocking resources Google needs to render the page: CSS, JavaScript, and images. A page Google cannot render properly is harder to understand and rank. Another classic is Disallow: / on a staging site that gets copied to production, silently blocking the entire live site. Remember that paths are case-sensitive, so Disallow: /Admin/ does not block /admin/. Trailing slashes matter too: Disallow: /blog blocks /blogging as well as /blog/, while Disallow: /blog/ blocks only what is under the folder. Finally, never list secret URLs in robots.txt hoping to hide them; the file is public, and attackers read it as a map of interesting paths. Test every change here before it goes live. Everything you paste stays in your browser.
Frequently Asked Questions
What does Disallow mean in robots.txt?
Disallow asks crawlers not to fetch any URL whose path starts with the given pattern. It is a request, not a lock: well-behaved bots like Googlebot obey it, but malicious scrapers ignore it completely. Never use robots.txt to protect genuinely sensitive content; that needs authentication.
Does robots.txt block indexing?
No, and this is the most misunderstood point in SEO. Robots.txt blocks crawling, not indexing. If other pages link to a blocked URL, Google can still index the URL itself and show it in results as a URL-only listing, without a title or description. To keep a page out of the index, let it be crawled and serve a noindex tag or header instead.
What does the $ do in robots.txt?
The dollar sign is an end-of-URL anchor. A rule like Disallow: /*.pdf$ matches any URL ending in .pdf, while Disallow: /*.pdf (without the dollar) also matches URLs like /file.pdf?download=1 because the pattern only needs to match the start of the path. Use $ when you want to match the exact ending of a URL.
How do I block one bot but allow all others?
Create a group naming that bot with Disallow: / and a second group for everyone else. For example: User-agent: BadBot, then Disallow: / on the next line, then a blank line, then User-agent: * with Disallow: (empty) or your normal rules. The named bot follows only its own group, and every other crawler follows the wildcard group.
Should I list my sitemap in robots.txt?
Yes, it is recommended. A line like Sitemap: https://example.com/sitemap.xml tells every crawler where your sitemap lives without any extra configuration. It does not replace submitting the sitemap in Google Search Console, which gives you indexing reports, but it guarantees discovery by any bot that reads your robots.txt.