What this tool does
Enter a domain or URL. The tool fetches /robots.txt from that site, shows the file, parses out the user-agent groups, the Disallow and Allow paths and the Sitemap lines, and flags lines with problems such as a missing colon or a directive it doesn’t recognise (with the line number).
What it tells you
- Whether the file exists and loads. A 404 means crawlers treat everything as allowed. A 5xx error can make some crawlers hold back, so a broken robots.txt is worth fixing quickly.
- Which sections apply to which crawlers.
- The sitemap addresses declared in the file.
- Syntax slips: “Line 7: Invalid syntax (missing colon)” or “Line 12: Unknown directive”.
What it doesn’t do
It checks syntax. It doesn’t test whether a particular URL is blocked for a particular crawler, and it doesn’t apply Google’s full precedence rules (the most specific rule wins, and Allow can override Disallow for a longer path). To test a URL against the rules, use the robots.txt report and URL Inspection in Google Search Console.
Mistakes worth catching
Disallow: /on a live site blocks everything. It is usually left over from staging.- Blocking CSS and JavaScript stops search engines from rendering your pages properly.
- Using robots.txt to hide pages. It doesn’t keep a page out of search results. A blocked page can still be indexed from links. Use
noindexinstead. - Putting private paths in the file. It’s public, so it advertises them.
To create one, use the robots.txt generator. Results are cached for 10 minutes, with up to 12 checks a minute.
Does it tell me whether a specific URL is blocked?
No. It validates and lists the rules. For URL-level testing use Google Search Console’s tools.
What happens if my site has no robots.txt?
Crawlers assume everything may be crawled. That is fine for most sites.
Does Disallow remove a page from Google?
No. It only stops crawling. Use noindex to keep a page out of results.
Why does it say Unknown directive?
The line isn’t a recognised robots.txt field, or it has a typo. Some crawlers support extras the tool doesn’t know.
Can I enter just a domain?
Yes. Enter a domain or a full URL and the tool fetches /robots.txt from that site.