Robots.txt Generator
iWebSeoTool Robots.txt Generator: Use this free robots.txt generator to build a robots txt file in seconds, with user-agent, allow and disallow rules, crawl-delay and a sitemap line you can download.
Use * for all search engines, or a name like Googlebot for one crawler.
Not every search engine uses crawl-delay.
Download the file and place it in your website's root folder, so it is at yourdomain.com/robots.txt. Everything runs in your browser — nothing is uploaded or stored.
How to use it
- Leave the user-agent as * for all crawlers, or choose a specific crawler.
- Add the paths you want crawlers to avoid under Disallow, and any exceptions under Allow.
- Add a crawl-delay and your sitemap address if you use them.
- Check the generated file, then copy or download robots.txt and upload it to your site root.
About this tool
This robots.txt generator works as a simple robots.txt creator: choose a user-agent, add a disallow rule for each path you want crawlers to skip, and add allow rules for exceptions inside a blocked folder. A crawl-delay line can ask crawlers to slow down, and a sitemap line points them to your sitemap so they can find your pages more easily.
The result is a plain robots txt file you can save and upload to your website root, at yourdomain.com/robots.txt. Remember that it guides well-behaved crawlers rather than blocking people, so use it for crawl guidance, not for hiding private pages. Like the other free tools from iWeb Seo Tool, it runs in your browser, so you can adjust a rule and regenerate the file without signing up. You can also create the tags for a page with the Meta Tag Generator or count your page text with the Word Counter.
Frequently asked questions
No. A robots.txt file only guides well-behaved crawlers; it does not hide a page from people, block direct visits, or provide security. Anyone who knows the address can still open it, so sensitive pages need proper access control instead of a robots.txt rule alone.
No. The rules you build are assembled in your browser and shown on the page for you to copy or download. They are not sent to a server for storage, so you can draft and adjust them without leaving a saved copy behind.
Yes. You can build, copy and download a robots.txt file here for free, with no sign-up and no payment needed. Just set your rules, check the result, and save the file for your site.
A Disallow rule tells crawlers not to crawl the path named after it, for example Disallow: /private/ asks crawlers to avoid everything under /private/. It is a request to well-behaved crawlers, and it is normally paired with a user-agent line that says which crawlers the rule applies to.
User-agent: * means the rules that follow apply to all crawlers. The star is a wildcard for every crawler name, so you do not need to list each search engine separately when the same guidance should apply to all of them.
A robots.txt file is a small plain-text file that tells search engine crawlers which parts of a website they may crawl and which parts they should avoid. It gives guidance through rules such as user-agent, allow, and disallow, and it can point crawlers to the site sitemap.
Put the file at the root of your website, so it is available at yourdomain.com/robots.txt. Crawlers look for it in that exact place, so a file in a subfolder will not be found as the site’s robots.txt.