robots.txt Guide: How to Write, Test & Optimize Your Robots File

A single misconfigured robots.txt can accidentally block your entire site from Google. A correctly written one saves crawl budget, protects admin pages, and helps search engines find your sitemap automatically. Here is everything you need to know to write one correctly.

What is robots.txt?

A robots.txt file is a plain text file placed at the root of your website (https://yourdomain.com/robots.txt) that follows the Robots Exclusion Protocol. It tells search engine crawlers — Googlebot, Bingbot, and others — which pages or sections they are allowed or not allowed to crawl. Most major crawlers respect these directives by default.

The file uses a simple syntax: one or more User-agent blocks, each containingAllow and Disallow directives. A Sitemap directive at the end points crawlers to your XML sitemap.

Core directives

User-agent specifies which crawler the rules apply to. Use * (wildcard) to target all crawlers. Specific User-agent rules take precedence over the wildcard — so you can block all crawlers from /admin/ while giving Googlebot a special exception.

Disallow blocks a path and everything under it. Disallow: /private/ prevents crawlers from accessing any URL starting with/private/. An empty Disallow: (no value) means allow everything — it is a common way to explicitly signal full access permission.

Allow overrides a broader Disallow for a specific path. The most common use is WordPress: Disallow: /wp-admin/ combined with Allow: /wp-admin/admin-ajax.php blocks the admin panel while allowing the AJAX endpoint that front-end plugins depend on.

Sitemap tells crawlers where to find your XML sitemap: Sitemap: https://example.com/sitemap.xml. Use the full absolute URL. Google, Bing, and most major crawlers support this directive.

Crawl-delay requests a delay (in seconds) between crawler requests. Useful for reducing server load from aggressive bots — but Googlebot ignores it. Control Googlebot crawl rate in Google Search Console instead. For other crawlers, 1–5 seconds is typical; above 10 seconds significantly slows indexing.

Platform templates

WordPress — block /wp-admin/ but allow /wp-admin/admin-ajax.php. Also block /wp-includes/ and /xmlrpc.php. Your sitemap is at /sitemap.xml (Yoast/RankMath) or /wp-sitemap.xml (WordPress core sitemaps).

Shopify — block /admin, /cart, /orders, and /checkouts/. Shopify auto-generates a robots.txt but you can extend it from the theme editor. Sitemap is at /sitemap.xml.

SaaS / web apps — block /api/, /admin/, /dashboard/, and /app/ to prevent crawlers from hitting authenticated routes. Allow your marketing pages, blog, and public documentation.

Common mistakes

Blocking CSS and JS: Old SEO advice said to block stylesheet and script directories. Modern Googlebot needs to render your pages to evaluate them — blocking CSS/JS prevents rendering and can hurt rankings. Never disallow these files.

Assuming robots.txt prevents indexing: Disallowing a URL prevents crawling, not indexing. Google can still index a URL it has never crawled if another site links to it. For true noindex, use a noindex meta tag or X-Robots-Tag HTTP header on the page itself.

No trailing slash on directory blocks: Disallow: /private only blocks the exact URL /private, not /private/page. Use Disallow: /private/ (with trailing slash) to block the directory and all URLs under it.

Blocking your own sitemap: If your sitemap is at /sitemap.xml and you have Disallow: /, crawlers cannot read the sitemap. Always verify your Sitemap URL is not covered by a Disallow rule.

Testing your robots.txt

Google Search Console includes a robots.txt Tester under the Legacy Tools section. Enter a URL from your site and it tells you whether Googlebot can access it under your current robots.txt rules. Bing Webmaster Tools has an equivalent tester.

For a quick structural check, use the validator built into the generator — it flags missing User-agent directives, overly broad blocks, and missing Sitemap URLs before you deploy.