Robots.txt Generator

Build a correct robots.txt file without memorizing the syntax. Add user-agent rules with allow and disallow paths, set an optional crawl delay, add your sitemap URL — and get a valid robots.txt ready to upload to your site’s root. Copy it or download it directly. The file is generated locally in your browser.

How to Use the Robots.txt Generator

1. Each row is one rule set: enter a user-agent (use * for all crawlers), an allow path, and/or a disallow path. Add more rows with + Add rule row.
2. Optionally set a crawl delay in seconds and paste your sitemap URL.
3. Click Generate robots.txt.
4. Copy or download the file and upload it as robots.txt in your website’s root folder.

Practical example

Your WordPress site’s /wp-admin/ area keeps showing up in search results. Add a row with user-agent * and disallow path /wp-admin/, add your sitemap URL, and generate the file. Upload it to your domain root, and well-behaved crawlers like Googlebot will stop requesting the admin area while still indexing everything else.

Key features and benefits

• Visual rule builder: no syntax to memorize — just fill in user-agents and paths.
• Multiple rule sets: different rules for different crawlers (for example, stricter limits for aggressive bots).
• Crawl delay support: politely slow down crawlers that hit your server too hard.
• Sitemap declaration: point crawlers at your sitemap automatically.
• Copy or download: get the file however suits your workflow.

Limitations

Robots.txt is a voluntary standard: well-behaved crawlers obey it, but malicious bots ignore it completely — it is not access control and must never be used to hide sensitive content. Overly broad disallow rules can accidentally block your whole site from Google, so double-check paths before uploading. Not all crawlers support crawl-delay. After uploading, verify the file is reachable at yourdomain.com/robots.txt and test it in Google Search Console, since a typo in one line can silently change how your entire site is crawled.