Robots.txt Generator

The Complete Robots.txt Generator Guide

How robots.txt files are structured, what each control in the generator does, and a worked example you can reproduce line by line.

Open Robots.txt Generator

What a Robots.txt File Does

A robots.txt file gives crawlers instructions about which paths on a host they may request. It is a plain text file served at the address /robots.txt, and its format follows the Robots Exclusion Protocol, standardized in RFC 9309. A crawler that follows the standard reads this file before it requests other URLs on the same host.

The file applies per host. The rules served at https://www.example.com/robots.txt do not cover https://blog.example.com, and an address with http counts as a separate host from its https version. If you run several hosts, each one needs its own file at its own root.

Robots.txt manages crawling, not indexing. A URL blocked by a Disallow rule can still appear in search results when other pages link to it, because the search engine can know the address without reading the page. To remove a page from results, the page must stay crawlable so the crawler can read a noindex instruction.

Step-by-Step Walkthrough

Work through the generator from top to bottom. Each step matches one part of the page.

  1. Pick a starting preset. Allow everything writes a group for * with one empty Disallow line, which explicitly allows all URLs. Block a private area writes two Disallow rules for example folders. Block the whole site writes Disallow: / for *. A preset replaces the groups you have entered but keeps your sitemap lines.
  2. Set the User-agent. Use an asterisk for every crawler that has no group of its own, or type a crawler name such as Googlebot. A crawler that matches a named group follows that group, and the asterisk group does not add its rules to it.
  3. Add Allow and Disallow rules. Each path should begin with a slash. A rule matches every URL whose path starts with its value, so /admin/ also covers /admin/login. When an Allow and a Disallow both match a URL, the longer and more specific rule wins. Paths are case-sensitive, so /Admin/ and /admin/ are different.
  4. Optionally set a Crawl-delay. Enter a whole number of seconds, or leave the field blank for no line. Google does not read this directive, and other crawlers support it to varying degrees.
  5. Add sitemap lines. Enter full https or http addresses, one per row. Sitemap lines are not tied to a group, so the output writes them after all the groups.
  6. Read the checks. An error means a line was left out of the file. A warning means the line was written but may not do what you intend. A note is information only.
  7. Copy or download. Download saves a file named robots.txt. Upload it to the root of the host so it is served at /robots.txt.

Errors leave a line out of the file, so read the preview before you copy it. Warnings and notes never stop a copy or a download, so review each one and decide what it means for your site.

Worked Example

Suppose a site wants to keep two folders away from general crawling, give Bing a slower pace, and publish one sitemap. The inputs are these rows:

  • Group 1: User-agent *, Disallow /admin/, Disallow /drafts/
  • Group 2: User-agent bingbot, Disallow /admin/, Crawl-delay 5
  • Sitemap row: https://www.example.com/sitemap.xml

The preview shows this file, and the summary reads 2 groups, 3 rules, 1 sitemap line:

User-agent: *
Disallow: /admin/
Disallow: /drafts/

User-agent: bingbot
Disallow: /admin/
Crawl-delay: 5

Sitemap: https://www.example.com/sitemap.xml

Bingbot reads only its own group, so /drafts/ is not blocked for it. If that folder should also be hidden from Bingbot, add a Disallow: /drafts/ row to group 2. The checks panel shows one note, that Google ignores Crawl-delay, because a group uses that line.

Now suppose the first rule had been typed as admin/ without the leading slash. The checks would show a warning that the path does not start with a slash, and the line would still be written as typed. You would fix it by changing the row to /admin/ and watching the warning disappear.

After you copy the file, compare it with the file already live at your address and test at least one URL you expect to block. The generator cannot test rules against your site, so that check is your responsibility.

Tips and Common Mistakes

  • Do not use robots.txt to hide private material. The file is public and lists the paths you want kept out, which tells curious readers where to look. Protect private areas with authentication.
  • Avoid blocking CSS or JavaScript that pages need in order to render. A search engine that cannot load those files may evaluate the page differently from what visitors see.
  • Check for Disallow: / before you publish. A stray line copied from a staging site can block an entire live site.
  • Keep one group per crawler. Two groups for the same name split the rules, and the checks will flag it.
  • Include a sitemap line. It is optional, but it gives crawlers a direct list of the URLs you want found.
  • Keep the file small. RFC 9309 asks crawlers to parse at least the first 500 kibibytes, so content past that point may be ignored.
  • Serve the file at the root with a normal success response. A missing or failing file is handled differently from one that loads cleanly.

Frequently Asked Questions

Where do I upload the robots.txt file?

Place it at the root of each host, so it is served at /robots.txt on that host. A subdomain such as blog.example.com needs its own file at its own root.

What is the difference between Disallow and noindex?

Disallow asks compliant crawlers not to fetch a URL. Noindex is a page-level instruction that tells search engines not to list the page, and a crawler must be able to fetch the page to see it. Use noindex when the goal is removal from results.

Are the rules case-sensitive?

Yes. Crawlers compare the path as written, so match the capital letters used in your real URLs.

Does this tool test my site?

No. It checks the text you enter for common formatting mistakes. It does not fetch your site and does not test how any crawler will read the file.

Open Robots.txt Generator