Robots.txt Generator
Create a focused robots.txt file without memorizing directive syntax.
Generated robots.txt
Your privacy is safe, this tool runs entirely in your browser. Nothing you enter is uploaded to or stored on our servers.
More SEO Tools
How to use Robots.txt Generator
- 1Choose the crawler or leave the user-agent as * for all crawlers.
- 2Add one allowed or disallowed path per line.
- 3Add the absolute URL of your XML sitemap.
- 4Copy or download robots.txt, then publish it at the root of your domain.
Build crawl rules that are easy to review
A robots.txt generator reduces syntax mistakes when you need to guide standards-compliant crawlers away from duplicate, private-looking or resource-heavy paths. Keep the file small and intentional.
Test important changes in Google Search Console before deployment. Blocking a page also prevents Google from seeing a noindex tag on that page, so crawl control and index control should be planned separately.
Frequently asked questions
Where should robots.txt be uploaded?
Publish it at the root of the hostname, such as https://example.com/robots.txt. A file inside a subfolder does not control the whole site.
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still appear if Google discovers links to it. Use an appropriate noindex response on crawlable pages when you need to prevent indexing.
Should CSS and JavaScript files be blocked?
Usually no. Search engines need page resources to render and understand the experience. Block only paths that crawlers genuinely should not request.
Can robots.txt protect private information?
No. The file is public and crawl rules are not access control. Protect private content with authentication and proper server authorization.
Can I list more than one sitemap?
Yes. Add one absolute sitemap URL per line. The generator creates a separate Sitemap directive for each valid entry.