Robots.txt Generator
Generate a robots.txt file to control how search engines crawl your site. Free, instant.
robots.txt
User-agent: * Allow: / Disallow: /admin Sitemap: https://yourdomain.com/sitemap.xml
What is robots.txt?
Robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages they can and cannot access. It is part of the Robots Exclusion Protocol and is respected by all major search engines including Google, Bing, and Yahoo.
Robots.txt directives
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: * |
| Allow | Explicitly allow crawling a path | Allow: /blog/ |
| Disallow | Block crawling a path | Disallow: /admin/ |
| Sitemap | Location of your XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests (Bing) | Crawl-delay: 10 |
Frequently Asked Questions
Where do I put robots.txt?
Place the file at the root of your domain: https://yourdomain.com/robots.txt. It must be accessible at this exact URL for crawlers to find it.
Does robots.txt block pages from appearing in Google?
No. Disallowing a page prevents crawling but not indexing. Google may still index the URL if other pages link to it. To prevent indexing, use a noindex meta tag instead.
Should I block AI bots with robots.txt?
If you don't want AI companies using your content for training, you can add specific user-agent rules for GPTBot, ChatGPT-User, and CCBot. Use the "Block AI Bots" preset above as a starting point.