Robots.txt Generator
Build a valid robots.txt file with templates for WordPress, Shopify, SaaS, and e-commerce. Add User-agent rules, Allow/Disallow paths, and Sitemap URL. Validate and download instantly — 100% private, runs in your browser.
Quick Templates
robots.txt preview
User-agent: * Sitemap: https://example.com/sitemap.xml
How to use the robots.txt generator
Start by clicking one of the Quick Templates — Blank, WordPress, Shopify, SaaS, or E-commerce — to pre-fill standard rules for your platform. Each template includes the most common Disallow paths for that type of site.
Customize the rules by editing the User-agent field (use * to target all crawlers, or a specific bot name like Googlebot). Add Disallow paths for sections you want to block and Allow paths for explicit exceptions. Use the + Add User-agent block button to create rules for specific crawlers.
Set your Sitemap URL in the Global Settings card — this adds a Sitemap: directive that helps search engines discover your sitemap automatically. Click Copy or Download to get the finished file.
robots.txt directives explained
User-agent specifies which crawler the rules apply to. Use * (wildcard) to target all crawlers, or a specific bot name like Googlebot, Bingbot, or GPTBot. Specific User-agent rules take precedence over the wildcard.
Disallow tells the crawler not to access that path or anything under it. Disallow: /admin/ blocks all URLs starting with /admin/. An empty Disallow: (no value) allows everything — it is a common way to explicitly tell crawlers they have full access.
Allow overrides a broader Disallow rule for a specific path. WordPress sites commonly use Allow: /wp-admin/admin-ajax.php alongside Disallow: /wp-admin/ to allow AJAX while blocking the admin panel.
Sitemap points to your XML sitemap. Supported by Google, Bing, and most major crawlers. Use the full absolute URL: Sitemap: https://example.com/sitemap.xml.
Crawl-delay requests a delay (in seconds) between crawler requests. Useful for reducing server load from aggressive bots. Note: Googlebot ignores this directive — use Google Search Console to control Googlebot crawl rate.
Platform-specific guidance
WordPress: Block /wp-admin/ but allow /wp-admin/admin-ajax.php (required for front-end AJAX features). Also block /wp-includes/ and /xmlrpc.php. Your sitemap is usually at /sitemap.xml (Yoast, RankMath) or /wp-sitemap.xml (WordPress core).
Shopify: Block /admin, /cart, /orders, and /checkouts/. Shopify auto-generates a robots.txt but you can customize it from the Online Store theme settings. The Sitemap is at /sitemap.xml.
SaaS / web apps: Block /api/, /admin/, /dashboard/, and /app/ to prevent crawlers from indexing authenticated-only pages. Allow your marketing pages and public documentation.
Features
- 5 quick templates — Blank, WordPress, Shopify, SaaS, E-commerce
- Multiple User-agent blocks — create rules for specific bots
- Allow and Disallow path management with add/remove controls
- Sitemap URL field and Crawl-delay support
- Live preview — robots.txt updates instantly as you type
- Validation — detects missing directives, overly broad blocks, and missing sitemaps
- Copy to clipboard and download as
robots.txt - 100% private — no data leaves your browser
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol standard and is respected by Google, Bing, and most major search engines.
Where do I place my robots.txt file?
The file must be in the root directory of your domain — accessible at https://yourdomain.com/robots.txt. It cannot be in a subdirectory. For WordPress, it goes in the same folder as wp-config.php. For Shopify and most hosted platforms, manage it through the platform's admin settings.
Does robots.txt prevent pages from appearing in Google search?
No. Blocking a URL with Disallow prevents Google from crawling it, but not from indexing it. If other sites link to the blocked page, Google may still index the URL (showing it in search results without a description). To prevent indexing, use a noindex meta tag or X-Robots-Tag response header.
What is the difference between Allow and Disallow?
Disallow blocks a path. Allow explicitly permits access, even within a broader Disallow rule — and takes precedence when both match. For example: Disallow: /admin/ combined with Allow: /admin/public/ lets crawlers access /admin/public/while blocking everything else under /admin/.
Should I add a Sitemap URL to my robots.txt?
Yes — it's strongly recommended. A Sitemap: directive (e.g. Sitemap: https://example.com/sitemap.xml) helps search engines discover your sitemap automatically without a separate submission. Google, Bing, and most major crawlers support this directive.
What is Crawl-delay and should I use it?
Crawl-delay asks crawlers to wait a set number of seconds between requests to reduce server load. However, Googlebot ignores it— use Google Search Console to control Googlebot's crawl rate instead. For other crawlers (Bingbot, etc.), values of 1–5 seconds are typical. Values over 10 seconds can significantly slow down indexing.