Robots.txt Syntax Guide
Complete reference for the robots.txt file. Every directive explained with practical examples.
Rather fill in a form? The robots.txt generator writes the file and tests it before you publish it.
What Is robots.txt?
The robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells search engine crawlers which pages or sections of your site they are allowed or not allowed to crawl.
It follows the Robots Exclusion Protocol, a standard respected by all major search engines including Google, Bing, and Yandex. It does not prevent pages from being indexed if they are linked from other sites. For that, you need noindex meta tags.
Basic Syntax
A robots.txt file consists of one or more groups. Each group starts with a User-agent line and contains one or more directives.
# This is a comment
User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xmlDirectives Reference
User-agent
Specifies which crawler the rules apply to. Use * to target all crawlers, or a specific bot name.
User-agent: *
Disallow: /admin/User-agent: Googlebot
Disallow: /no-google/
User-agent: Bingbot
Disallow: /no-bing/Disallow
Tells the crawler not to access the specified path. An empty Disallow: means nothing is disallowed.
User-agent: *
Disallow: /admin/
Disallow: /tmp/
Disallow: /api/Allow
Overrides a Disallow for a specific sub-path. Useful for allowing a page inside a blocked directory.
User-agent: *
Disallow: /private/
Allow: /private/public-page.htmlSitemap
Points crawlers to your XML sitemap. This is independent of any User-agent group and can appear anywhere in the file. You can list multiple sitemaps.
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/blog-sitemap.xmlCrawl-delay
Specifies the number of seconds a crawler should wait between requests. Supported by Bing and Yandex. Google ignores this directive, and it retired its crawl rate setting in Search Console in January 2024. To slow Googlebot down for a while, Google recommends answering with 500, 503 or 429 status codes.
User-agent: Bingbot
Crawl-delay: 10Common Patterns
Block All Crawlers
User-agent: *
Disallow: /Use this for staging or development environments that should never be indexed.
Allow All Crawlers
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAn empty Disallow means everything is accessible. Always include your sitemap.
Block Specific Paths
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search?
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlBlock admin pages, API endpoints, and URL parameters that create duplicate content.
Block Specific Bots
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlBlock AI training crawlers while allowing search engines to access your content.
Common Mistakes
Blocking CSS and JavaScript
If you Disallow /css/ or /js/, Googlebot can't render your page properly. This hurts your rankings. Never block static assets.
# DON'T do this:
User-agent: *
Disallow: /css/
Disallow: /js/Forgetting the Sitemap directive
The Sitemap line helps crawlers discover all your pages. Always include it, even if you've already submitted it in Search Console.
Using robots.txt to hide pages from Google
Disallow prevents crawling, not indexing. If other sites link to a blocked page, Google can still index the URL (without content). Use a noindex meta tag instead.
Case sensitivity issues
Paths in robots.txt are case-sensitive. /Admin/ and /admin/ are treated as different paths. Double-check your casing.
Wrong file location
robots.txt must be at the root of your domain: https://example.com/robots.txt. It won't work in a subdirectory.
How Crawlers Match Rules
A crawler first picks its group: the group whose User-agent names it, matched without regard to case, or the * group when none does. Several User-agent lines in a row share the rules below them, and two groups naming the same crawler are combined. Googlebot-News with no group of its own follows the Googlebot group before the * group.
Inside that group, the longest matching path wins, whatever the order of the lines, and when an Allow and a Disallow are the same length, Allow wins. * matches any sequence of characters and $ marks the end of the URL. Paths are case-sensitive.
| Rules | URL | Result |
|---|---|---|
| Allow: /p · Disallow: / | /page | Allowed: /p is longer |
| Allow: /folder · Disallow: /folder | /folder/page | Allowed: a tie goes to Allow |
| Allow: /page · Disallow: /*.htm | /page.htm | Blocked: /*.htm is longer |
| Allow: /$ · Disallow: / | /page.htm | Blocked: /$ matches only the home page |
| Disallow: /*.php$ | /index.php?id=1 | Allowed: the query comes after .php |
| Disallow: /fish | /Fish.asp | Allowed: paths are case-sensitive |
Test any URL against your own file, for Googlebot, Bingbot or an AI crawler, with the robots.txt tester.
Status Codes, Size and Caching
- 404 or another 4xx: crawlers treat the site as having no restrictions and may fetch everything (RFC 9309).
- 5xx or no answer: crawlers must treat the whole site as disallowed until the file answers again, so a broken robots.txt can stop crawling entirely.
- Size: Google reads the first 500 KiB and ignores the rest.
- Caching: Google generally caches robots.txt for up to 24 hours, so a change is not seen at once.
- Not supported by Google: Crawl-delay, and noindex inside robots.txt, which Google stopped honouring in 2019. Unknown lines, including typos such as Disalow, are silently ignored by every crawler.
Complete Example Template
Here's a production-ready robots.txt template you can adapt for your website:
# Robots.txt for example.com
# Allow all search engines
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Disallow: /*?utm_
Disallow: /*?ref=
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Sitemaps
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/blog-sitemap.xmlCheck Your Robots.txt
Validate your robots.txt file for syntax errors and misconfigurations.