What this tool does
Pick the crawler the rules apply to, list the paths to block or allow (one per line), add your sitemap URL, and the tool writes the file as you type. Copy it into a plain text file named robots.txt and upload it to the root of your site, so it is reachable at https://yoursite.com/robots.txt.
The crawler list includes all crawlers (*), Googlebot, Googlebot-Image, Bingbot, and three AI crawlers: GPTBot, ClaudeBot and Bytespider.
How to use it
- Choose the crawler the rules apply to, or press a preset.
- Type the paths to block under Disallowed Paths, one per line. Add exceptions under Allowed Paths.
- Add your XML sitemap URL if you have one.
- Press Copy or Download, and upload the file to the root of your site.
- Test it with the robots.txt checker.
What a robots.txt file can and can’t do
It is a request to well-behaved crawlers about what to fetch. It does not keep anything private, and it does not remove pages from search results. A blocked URL can still be indexed if other sites link to it, and it will show up without a description. To keep a page out of search, use a noindex meta tag or an X-Robots-Tag header, and leave the page crawlable so the crawler can see that instruction.
Malicious bots ignore robots.txt. Don’t list sensitive paths in it, because the file is public and tells everyone where to look.
Using the generator
- Paths should start with
/. The tool adds the slash if you forget. - This generator writes one group of rules for the crawler you choose. If you want different rules for Googlebot and for AI crawlers, generate each one separately and put the groups together in one file.
Disallow:with nothing after it means “allow everything”.- Rules are matched by prefix, and for Google the most specific matching rule wins.
For a WordPress site
A common starting point is to block /wp-admin/ while allowing /wp-admin/admin-ajax.php, which front-end features often call. Don’t block your CSS and JavaScript folders: search engines need them to render your pages.
Test the final file with the robots.txt checker after you upload it.
Will a Disallow rule remove a page from Google?
No. It stops crawling, not indexing. Use a noindex tag to keep a page out of results, and let it be crawled so the tag can be seen.
Can I set a crawl delay?
This generator doesn’t write one. Googlebot ignores the Crawl-delay directive anyway, although some other crawlers such as Bingbot honour it.
Can I block AI crawlers with this?
You can generate a rule for GPTBot, ClaudeBot or Bytespider. Whether a crawler obeys robots.txt is up to the crawler.
Can it make rules for several crawlers in one file?
One group at a time. Generate each group and combine them in your file.
Where do I put the robots.txt file?
In the root of the site, so it loads at https://yoursite.com/robots.txt. A file in a subfolder is not read by crawlers, and each host name, including subdomains, needs its own file.