Robots.txt Generator
Create a robots.txt file that allows or blocks crawlers, disallows paths, lists your sitemaps and can block AI training crawlers or AI search bots, ready to copy or download.
Input
Output
Readme
What is robots.txt?
A robots.txt file sits at the root of a website, as in https://example.com/robots.txt, and tells crawlers which pages they may fetch. Search engines and other well-behaved bots read it before they crawl a site. It isn't access control: anyone can still open the pages, and a bot that ignores the file isn't stopped by it.
How to use it
- All crawlers: choose Allow all or Block all. With Allow all, list the paths crawlers should stay out of under Disallowed paths, one per line, like /admin/ or /cart. Allowed paths reopen paths inside them, like /admin/help/. With Block all, Allowed paths are the only places crawlers may go.
- AI crawlers: two checkboxes add groups that keep AI crawlers out of the whole site (see below).
- Other: Crawl delay asks crawlers to wait that many seconds between requests. Bing and Yandex follow it, Google ignores it. Sitemap URLs list your sitemaps by their full address, like https://example.com/sitemap.xml.
The file updates as you type. Copy it, or download robots.txt and upload it to the root folder of your site.
Writing paths
- A path starts with a slash and matches every address that begins with it: /admin matches /admin, /admin/ and /administrator, while /admin/ matches only the folder and what's inside it.
- An asterisk matches any characters and a dollar sign marks the end of an address, so /*.pdf$ matches every PDF file. Google, Bing and most other major crawlers understand both.
- Paths are case-sensitive: /Admin/ and /admin/ are different.
- Lines that aren't paths are left out of the file and listed above it, so you can fix them.
How the rules work together
A crawler follows the group that names it most specifically, and only that group. A crawler named in one of the AI groups ignores the rules for all crawlers and is shut out entirely. Inside a group the longest matching path wins, so Allow: /admin/help/ opens that folder within Disallow: /admin/. The generator writes Allow lines before Disallow lines, which gives the same result for older crawlers that take the first rule that matches.
Blocking AI crawlers
Block AI training crawlers shuts out the crawlers that collect pages for training AI models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training and grounding at Google; Google Search isn't affected), Applebot-Extended (Apple's AI models; Apple's search isn't affected), CCBot (Common Crawl, a public dataset many AI models are trained on), meta-externalagent (Meta) and Amazonbot (Amazon).
Block AI search and assistant bots shuts out the bots that read pages for AI search results and for assistants answering someone's question: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity), meta-webindexer and meta-externalfetcher (Meta), and Amzn-SearchBot and Amzn-User (Amazon). Blocking them can keep your pages out of those AI answers and the links in them. Bots that fetch a page because a person asked for it may not follow robots.txt, their operators say.
Every name is the user-agent token its operator documents for robots.txt.
Example
With /admin/ disallowed, /admin/help/ allowed, the AI training crawlers blocked and a sitemap, the file reads:
User-agent: *
Allow: /admin/help/
Disallow: /admin/
# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
...
Disallow: /
Sitemap: https://example.com/sitemap.xmlFeatures
- Allow or block all crawlers, with disallowed and allowed paths
- Block AI training crawlers, or AI search and assistant bots, by their documented names
- Crawl delay and sitemaps
- Invalid paths and addresses flagged
- Copy or download robots.txt