tigtools

Robots.txt Generator

Choose what crawlers may fetch, add rules for specific bots, and download a valid robots.txt for your site's root.

Runs in your browser No sign-up

Presets
All crawlers (User-agent: *)
Default policy

Each path must start with / or *. Use $ to match the end of a URL, for example /*.pdf$.

Google ignores Crawl-delay; some other crawlers respect it.

Rules for specific crawlers

Adds a group that disallows GPTBot, CCBot, Google-Extended, ClaudeBot and PerplexityBot. Compliance with robots.txt is voluntary.

robots.txt is public: anyone can read it, and it can reveal the paths you list. It only asks well-behaved crawlers not to fetch pages; it does not protect anything. Use authentication for private content, and noindex for pages that should stay out of search results.

How to use it

  1. Choose the default policy for all crawlers and add any paths they should not fetch, such as /cart/ or /search.
  2. Add groups for specific crawlers if they need different rules, or use a preset.
  3. Add the full URL of your XML sitemap.
  4. Download robots.txt and upload it to the root of your site, so it is served at https://your-domain/robots.txt.

How it works

robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may fetch. It is standardised as RFC 9309. A file contains groups: one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the group that matches its name most specifically, and falls back to the * group otherwise; it does not combine its own group with the * group. Within a group, the most specific (longest) matching rule wins, and Allow wins a tie.

Paths are matched from the start of the URL path and are case-sensitive. * matches any sequence of characters and $ marks the end of the URL. An empty “Disallow:” means nothing is blocked. Lines in this tool that do not start with / or * are ignored and flagged, because crawlers would ignore them too. Sitemap lines must be absolute URLs and apply to the whole file, not to one group.

Important limits: robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, just without a description. To keep a page out of search results, allow crawling and use a noindex robots meta tag instead. Each host and protocol needs its own file, so blog.example.com needs a separate robots.txt.

Examples

Shop that hides cart and internal search

Allow everything, then disallow /cart/, /checkout/ and /search. Add Sitemap: https://example.com/sitemap.xml. Product and category pages stay crawlable.

Staging site

Choose “Disallow everything”. Well-behaved crawlers then skip the site, but protect staging with a password as well, because robots.txt is only a request.

Opting out of AI training crawlers

The AI crawler preset adds a group for GPTBot, CCBot, Google-Extended, ClaudeBot and PerplexityBot with Disallow: /. Search crawlers such as Googlebot keep following the * group.

Frequently asked questions

Where do I put robots.txt?

In the root of the host, so it loads at https://example.com/robots.txt. A robots.txt in a subfolder is ignored.

Will Disallow remove a page from search results?

Not reliably. Disallow stops crawling, but the URL can still be listed if it is linked from elsewhere. Use a noindex meta tag or header, and keep the page crawlable so the crawler can see it.

Does every bot obey robots.txt?

No. Following robots.txt is voluntary. Major search engines and many AI crawlers state that they respect it, but scrapers and malicious bots can ignore it.

Does Google use Crawl-delay?

No, Google ignores Crawl-delay. Some other crawlers, including Bing's, read it. Google adjusts its crawl rate automatically based on how your server responds.

Should I block CSS and JavaScript files?

No. Search engines render pages and need those files to understand the layout and content. Blocking them can make pages look broken to crawlers.

Privacy

The file is generated in your browser. Your rules and URLs are not sent anywhere or stored.