ValidateHTML
Your sites

Robots.txt Generator

Say what crawlers may read, and copy a robots.txt that does exactly that. Every file is tested before you publish it.

By default, search engines and other crawlers may

One per line. A path matches everything that starts with it; * matches any characters and $ marks the end.

Block these AI crawlers from the whole site

Your robots.txt

User-agent: *
Allow: /

Upload it to the root of the site, so it answers at https://your-site/robots.txt.

Test this file before you publish it

Allowed by Allow: / on line 2

What the Generator Checks

Read back by a real parser

The file is parsed with the same RFC 9309 engine as our robots.txt checker, and every path, exception and blocked crawler is tested against it.

Rules that would not work

An Allow that loses to a longer Disallow, a Disallow cancelled by an equal Allow: you are told before you upload, not after a crawl.

Nothing injected

A path containing a line break or a space is refused, so a paste can never add a directive you did not write.

A tester built in

Type any path, pick a crawler, and see whether it is allowed and which line decides it, using Google's longest-match rule.

How the Rules Combine

A crawler obeys only the group that names it most specifically. If the file has a group for GPTBot, GPTBot ignores the User-agent: * group entirely, which is why the generator gives blocked crawlers a group of their own with Disallow: / and leaves everyone else on the shared rules.

Inside a group, the rule with the longest matching path wins, and Allow wins a tie. Disallow: /admin/ with Allow: /admin/help/ keeps crawlers out of the admin area except the help pages. A * matches any run of characters and a final $ anchors the end, so /*.pdf$ blocks PDF files and nothing else. The robots.txt syntax guide walks through every directive, and the robots.txt checker tests the file once it is live.

Frequently Asked Questions

Where do I put the robots.txt file?
At the root of the host it applies to, so it answers at https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain (shop.example.com, blog.example.com) needs its own. The name is lowercase and the file is plain text in UTF-8.
Does blocking a page in robots.txt remove it from Google?
No. Disallow stops crawling, not indexing: a blocked URL that other pages link to can still appear in results, without a description. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or an X-Robots-Tag header. Google stopped honouring noindex inside robots.txt in 2019.
Should I block AI crawlers?
It is a business decision, and the file can only express it for crawlers that read robots.txt. Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) and the controls Google-Extended and Applebot-Extended decide whether your content is used to train models; blocking them has no effect on Google or Bing search. Search crawlers for AI products (OAI-SearchBot, PerplexityBot) decide whether you can appear in those answers, so blocking them also removes that traffic. Each name is a separate token: blocking GPTBot does not block OAI-SearchBot.
Why does the generator say an Allow has no effect?
When two rules match a URL, the longest one wins, and Allow wins a tie (RFC 9309, as Google applies it). An Allow shorter than the Disallow it was meant to override therefore loses. The generator reads every file back with the same rules as the robots.txt checker and tells you when a line would not do what you asked.
Can robots.txt protect private pages?
No. The file is public and only polite crawlers obey it; listing a secret path there even tells everyone where to look. Protect private areas with authentication, and keep staging sites behind a password as well as a Disallow: /.
Is Crawl-delay supported?
Google ignores it; Bing and some other crawlers honour it. The generator leaves it out, because a delay that the main search engine ignores tends to give a false sense of control. Google's crawl rate adjusts automatically to how fast your server answers.
How big can robots.txt be?
Google reads the first 500 KiB and ignores the rest. A generated file is a few hundred bytes; long files usually come from listing URLs one by one, which a wildcard pattern such as /*?sort= replaces.

Related Tools

Robots.txt CheckerRobots.txt SyntaxMeta Tag CheckerHTTP Status Checker