RFC 9309 Compliant • 100% Free SEO Utility

Online Robots.txt Generator & Validator

Generate clean, error-free robots.txt files for search engine web crawlers. Specify custom allow and disallow paths, configure crawl-delays, and declare XML sitemaps to optimize your site's crawl budget.

User Agents Configuration
Quickly apply pre-configured settings for specific crawlers
Specify which crawler these rules apply to
Delay between crawler requests in seconds
Paths you want to block from crawling
Paths you explicitly want to allow (overrides Disallow)
Location of your sitemap file
Test URL Access
1 lines
User-agent: *
Download File

Live Access Preview

User-agent//cart/checkout/blog
*Allowed ✅Allowed ✅Allowed ✅Allowed ✅

Comprehensive Guide to Robots.txt: Syntax, Crawl Budgets & Technical SEO Best Practices

Every major search engine begins its discovery lifecycle by requesting your website's /robots.txt file. Standardized under RFC 9309 (Robots Exclusion Protocol), this lightweight plain-text file acts as the primary gatekeeper between your web server and automated crawler bots, instructing them on which server resources they are authorized to request and crawl.

A well-crafted robots.txt file preserves precious server CPU cycles, eliminates crawl traps, prevents indexing of duplicate administrative screens, and accelerates the discovery of newly published editorial articles. Conversely, a single misplaced slash (e.g. Disallow: /) can de-index an entire million-dollar online enterprise from Google search overnight.

Core Directives of the Robots Exclusion Protocol

Robots.txt syntax is structured into distinct rule blocks identified by user-agent headers:

DirectiveSyntax ExampleFunction & SEO Impact
User-agentUser-agent: GooglebotIdentifies which specific crawler the subsequent rules apply to. Asterisk * denotes all compliant web bots.
DisallowDisallow: /admin/Forbids the targeted bot from requesting any URL starting with that path prefix. Leaving it empty (Disallow:) permits all access.
AllowAllow: /admin/loginOverrides a broader Disallow rule, permitting access to a specific subpath within an otherwise blocked directory.
SitemapSitemap: https://example.com/sitemap.xmlDeclares the absolute HTTPS URL of your XML sitemap, allowing search engines to discover all canonical routes immediately.
Crawl-delayCrawl-delay: 5Requests that bots (like Bingbot and Yandex) wait the specified number of seconds between requests to avoid server CPU overload.

Mastering Wildcard Pattern Matching (* and $)

Modern search engines support flexible pattern matching using wildcard operators:

The Asterisk (*) Wildcard

Matches any sequence of zero or more characters.

Disallow: /*?sort=*
Disallow: /*/temp/*Blocks all faceted sorting URLs containing query strings without impacting clean canonical URLs.
The Dollar Sign ($) End-of-URL Anchor

Asserts that the match must occur at the very end of the URL path.

Disallow: /*.pdf$
Disallow: /*.zip$Blocks direct crawling of raw PDF downloads without blocking landing pages like /pdf-tools/.

Crawl Budget Optimization for High-Scale Websites

Google does not have unlimited time to spend crawling your server. Crawl Budget is the number of URLs Googlebot can and wants to crawl on your site during a given timeframe. If your website has 50,000 URLs, but 35,000 are duplicate faceted filter permutations (e.g. ?color=blue&size=xl&view=grid), Googlebot may exhaust its crawl budget on duplicate filters and fail to discover newly published blog posts or products.

By adding targeted Disallow directives for filter parameters, internal search results pages (Disallow: /search?*), and session-tracking parameters, you funnel 100% of the crawler's attention toward your high-value canonical pages.

8 Critical Robots.txt Mistakes to Avoid

  1. Blocking CSS and JavaScript: Never write Disallow: /css/ or Disallow: /_next/. Google must render scripts to evaluate responsive design and content layout.
  2. Using Robots.txt for Sensitive Security: Robots.txt is a publicly viewable file. Never list secret admin URLs, internal staging paths, or unreleased product URLs in it, as malicious bots scrape robots.txt to discover hidden attack targets. Use HTTP authentication instead.
  3. Confusing Disallow with Noindex: Disallow stops crawling, not indexing. If an external backlink points to a disallowed URL, Google will still index the empty URL stub. To de-index, use <meta name="robots" content="noindex" />.
  4. Incorrect File Location: The file must be accessible at https://yourdomain.com/robots.txt. A subfolder path like /assets/robots.txt is completely ignored.
  5. Wrong Casing: Robots.txt directives and URL paths are case-sensitive. Disallow: /admin/ does not block /Admin/.
  6. Leaving Staging Rules in Production: Deploying a staging robots.txt containing Disallow: / to production will wipe your entire organic search traffic within days.
  7. Relative Sitemap URLs: Always specify the full absolute URL for sitemaps including protocol (e.g. https://www.example.com/sitemap.xml).
  8. Missing File Encoding: Robots.txt must be saved as standard UTF-8 without Byte Order Mark (BOM).

Frequently Asked Questions (Robots.txt FAQs)

A robots.txt file is a plain text file placed at the root directory of your web server (e.g. https://www.yourdomain.com/robots.txt). It operates under the Robots Exclusion Protocol (formalized in RFC 9309) to instruct automated search engine web crawlers (such as Googlebot, Bingbot, and Applebot) which paths, directories, or parameters they are permitted or forbidden to request and crawl.

This is one of the most critical distinctions in SEO. A 'Disallow' rule in robots.txt prevents search bots from fetching and crawling the page's HTML contents. However, if external websites link to that disallowed URL, Google can still index the URL and display it in search results with a snippet saying 'No information is available for this page'. To completely prevent a page from appearing in search results, allow it to be crawled in robots.txt and add an on-page meta tag '<meta name="robots" content="noindex, follow" />'.

Modern search engines like Google render web pages as a headless browser to understand visual layout, mobile responsiveness, and dynamic content. If your robots.txt file blocks access to CSS stylesheets, JavaScript bundles, or image assets, Googlebot cannot render the DOM properly, resulting in severe search ranking penalties and mobile usability errors.

The asterisk wildcard '*' acts as a catch-all rule applying to all automated search engine robots and scrapers that adhere to the Robots Exclusion Protocol. If a specific bot (e.g. 'User-agent: Googlebot') has its own dedicated rule block elsewhere in the file, it will follow its dedicated block instead of the generic '*' catch-all.

Google and Bing support pattern matching in path directives. The asterisk '*' matches any sequence of zero or more characters (e.g. 'Disallow: /*?sort=' blocks all parameterized sort queries). The dollar sign '$' matches the exact end of a URL (e.g. 'Disallow: /*.pdf$' blocks all URLs ending strictly with '.pdf' without affecting paths like '/pdf-tools/').

No. Googlebot ignores the 'Crawl-delay' directive entirely; instead, crawl rate can be calibrated inside Google Search Console or is managed automatically by Google's adaptive algorithms. However, other search crawlers—such as Bingbot, Yandex, and Baidu—respect and enforce the 'Crawl-delay' directive (in seconds) to avoid overloading web server resources.

The robots.txt file must reside strictly in the top-level root directory of your domain: 'https://example.com/robots.txt'. Placing it inside a subfolder like 'https://example.com/public/robots.txt' will cause search crawlers to ignore it completely, as bots only check the root path.

Search engines allocate a finite 'crawl budget' (number of pages fetched per day) to each domain based on server response speed and site popularity. On large websites (e-commerce catalogs, forums, faceted search portals), thousands of duplicate filtered URLs (e.g. '?color=red&size=m&sort=price') consume crawl budget, preventing bots from discovering high-value product and article pages. Disallowing infinite faceted filters in robots.txt directs crawlers straight to your revenue-generating content.

Explore Related Tools

Hand-picked utilities and calculators related to this tool.

SEO Tools
Robots Meta Tag & X-Robots Generator
→
SEO Tools
Schema Markup (JSON-LD) Generator
→
SEO Tools
SEO Friendly URL & Slug Generator
→
SEO Tools
XML Sitemap Generator & Validator
→
SEO Tools
HTML Meta Tag Generator
→
SEO Tools
Open Graph & Social Media Card Generator
→
SEO Tools