Free SEO Tool
Robots.txt Tester
Paste a robots.txt and a list of URLs, and get a verdict per URL: allowed or blocked, plus the exact rule and line number that decided it. The matcher follows Google's documented behaviour — most specific user-agent group wins, longest rule path wins, and on a tie the least restrictive rule wins — so a surprising result here is a surprising result in production.
How it works, plainly: this is a rule-based generator, not an AI model. Every result is assembled from tested formulas, templates and validators that run entirely in your browser — no model call, no cost, and nothing you type is uploaded.
- Free
- No signup
- Shows the deciding rule
- Runs in your browser
How it works
- 01 Paste your robots.txt
The live file from yourdomain.com/robots.txt, or a draft you have not shipped yet.
- 02 List the URLs to test
Full URLs or paths, one per line. Query strings are matched too — they are part of the path a crawler compares.
- 03 Pick the crawler
Googlebot, Googlebot-Image, Bingbot, GPTBot and others, or type any user-agent token. Group selection changes the answer.
- 04 Read the deciding rule
Each row names the rule and line that won, so you can fix the file rather than guess at it.
Next step
FAQ
Frequently asked questions
- Which rule wins when allow and disallow both match?
- Google picks the most specific rule, measured by the length of the rule path. If two rules are the same length, the least restrictive one wins — which means allow beats disallow on a tie. Google's own examples: for https://example.com/folder/page with 'allow: /folder' and 'disallow: /folder', the allow applies because the rules conflict and allow is less restrictive; for https://example.com/page.htm with 'allow: /page' and 'disallow: /*.htm', the disallow applies because its rule path is longer and matches more characters. This tester implements exactly that, and every one of Google's six published precedence examples is covered by the test suite behind this page.
- Does robots.txt keep a page out of Google?
- No, and this is the most expensive robots.txt misunderstanding. Disallow stops a compliant crawler from fetching the URL; it does not stop the URL being indexed if other pages link to it. Worse, because the page is never fetched, a noindex rule on it is never read: Google's documentation states that if a page is disallowed from crawling through robots.txt, any information about indexing or serving rules will not be found and will therefore be ignored. To keep a page out of search, leave it crawlable and serve a noindex robots meta tag or X-Robots-Tag header.
- Does Google support crawl-delay?
- No. Google supports exactly four fields in robots.txt — user-agent, allow, disallow and sitemap — and its documentation names crawl-delay as an example of a field that is not supported. The line is not an error; it is simply skipped, and skipping it does not break the group it sits in. Bing and Yandex do read crawl-delay, so leaving it in is a reasonable choice for those crawlers. This tester flags it so you know Googlebot is ignoring it.
- How do user-agent groups get chosen?
- Only one group applies to a given crawler: the one with the most specific matching user-agent. Non-matching text is ignored, so googlebot/1.2 and googlebot* are both equivalent to googlebot. If the same user-agent heads several groups, their rules are merged internally into one. A specific group and the wildcard * group are never combined — if any named group matches, the * group is discarded entirely for that crawler. That last rule catches people out: rules you put under * stop applying the moment you add a Googlebot section.
- Is there a size limit on robots.txt?
- Yes. Google enforces a file size limit of 500 kibibytes and ignores everything after it. If you are near that, consolidate rules — Google's own advice is to move excluded material into a separate directory so one rule covers it. This tester reports the byte size of what you paste and warns above the limit.
- Are the paths case-sensitive?
- The path value is case-sensitive; the field names and the user-agent value are not. So 'disallow: /fish' blocks /fish.html and /fishheads but not /Fish.asp, and 'Disallow' and 'disallow' behave identically. Wildcards are limited to two characters: * matches zero or more of any character, and $ anchors to the end of the URL.