Robots.txt Checker and Tester
Enter a site or paste a file to validate its robots.txt, then test whether any URL is allowed or blocked for Googlebot, Bingbot or an AI crawler, and which line decides it.
Writing the file yourself? The robots.txt generator builds one from a form and tests it, and the robots.txt syntax guide covers every directive, common patterns and the most frequent mistakes.
What Is Robots.txt?
The robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections of your site they can or cannot access.
It follows the Robots Exclusion Protocol, a standard used by all major search engines including Google, Bing, and Yahoo. While robots.txt is not a security mechanism, being advisory rather than enforceable, well-behaved crawlers will respect its directives.
A properly configured robots.txt file helps you manage your crawl budget, prevent indexing of duplicate or private content, and point crawlers to your XML sitemaps. Misconfigurations can accidentally block important pages from search engines, causing significant SEO damage.
Essential Robots.txt Directives
User-agent
Specifies which crawler the following rules apply to. Use * (asterisk) to target all crawlers, or specify individual bots like Googlebot, Bingbot, or GPTBot.
User-agent: *Disallow
Tells the crawler not to access a specific URL path or directory. An empty Disallow value means nothing is blocked for that user-agent.
Disallow: /admin/Allow
Explicitly permits access to a path within a disallowed directory. Useful for making exceptions. Supported by Google and Bing.
Allow: /admin/public/Sitemap
Points crawlers to your XML sitemap. Unlike other directives, Sitemap is not tied to any user-agent and applies globally. You can list multiple sitemaps.
Sitemap: https://example.com/sitemap.xmlCrawl-delay
Requests that the crawler wait a specified number of seconds between requests. Supported by Bing and Yandex but ignored by Google (use Search Console instead).
Crawl-delay: 10Common Robots.txt Mistakes
Blocking CSS and JS Files
Blocking CSS or JavaScript files prevents search engines from rendering your page, which hurts rankings. Google needs access to these resources to evaluate your content properly.
Blocking the Entire Site
A single Disallow: / under User-agent: * blocks all crawlers from your entire site. This is sometimes left in place accidentally after a staging or pre-launch phase.
Missing Sitemap Declaration
Not including a Sitemap directive means crawlers have to discover your sitemap through other means. Always declare your sitemaps in robots.txt for faster discovery.
Using Robots.txt for Privacy
Robots.txt is publicly readable and only advisory. Never rely on it to hide sensitive pages. Use authentication, meta noindex tags, or HTTP headers instead.
Incorrect Path Syntax
Paths in robots.txt are case-sensitive and must start with a forward slash. A typo or missing slash can silently invalidate your rule.
Wrong File Location
The robots.txt file must be at the root of your domain. Placing it in a subdirectory (e.g., /blog/robots.txt) will be ignored by crawlers.
What This Robots.txt Validator Reports
Typos crawlers ignore
Disalow: /admin is not a rule: every crawler skips the line. You get the line number and the directive it was probably meant to be.
Rules that apply to no one
An Allow or Disallow written before any User-agent line belongs to no group, so nothing obeys it.
Paths that match nothing
Disallow: admin, without the leading slash, never matches a URL.
Directives Google ignores
Crawl-delay is ignored by Google, and noindex in robots.txt stopped working in 2019. Both are reported with what to use instead.
A site closed to crawlers
Disallow: / under User-agent: * keeps every crawler without its own group out of the whole site. Right for staging, a disaster in production.
Size and status
Google ignores everything past 500 KiB, and a robots.txt that answers with a server error makes crawlers stop crawling the site.
Test a URL Before You Change Anything
After the check, the tester tells you whether a given URL is allowed or blocked for Googlebot, Bingbot, GPTBot, Google-Extended, ClaudeBot or any other crawler, and which line decides it. It applies the rules the way Google documents them: the crawler's own group or the * group, the longest matching path, Allow on a tie, * and $ wildcards.
To test a file before it goes live, choose Paste a file: nothing is fetched, so you can try a rule on a staging copy and see its effect on real URLs first. For the full rules with examples, read the robots.txt guide.