121 · Security Tools

Robots.txt Generator

Choose what crawlers may visit and download a ready-made robots.txt for your site’s root folder.

Search engines

Start each path with /. Leave empty to block nothing.

Google ignores this; Bing and others respect it.

How to use the robots.txt generator

  1. Allow or block everything, then list paths to block.
  2. Add your sitemap URL.
  3. Download robots.txt and upload it to your site’s root, e.g. example.com/robots.txt.

What robots.txt can and can’t do

robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it doesn’t hide pages: anyone can still open a blocked URL, and blocked pages can still appear in search results if other sites link to them. To keep a page out of search results, use a noindex meta tag instead (see the meta tag generator). Never list secret URLs in robots.txt.

Examples

Block /admin/ and add a sitemap

User-agent: * Disallow: /admin/ Sitemap: https://…

Ask AI crawlers not to crawl

Adds Disallow rules for GPTBot, ClaudeBot, CCBot and others

Questions

Where do I put the file?

In the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own.