robots.txt Generator
Build a robots.txt from presets or custom rules: choose user-agents, add Allow/Disallow paths, set crawl-delay and sitemap URLs, and optionally block AI training crawlers like GPTBot. Output updates live and the page flags the classic empty-Disallow trap. Fully generated in your browser.
* means every crawler not named elsewhere. Once a bot is named, it follows only its own group.
An empty path means no restriction; Disallow: / is what blocks everything. * and $ are supported.
About this tool
Build a robots.txt from presets or custom rules: choose user-agents, add Allow/Disallow paths, set crawl-delay and sitemap URLs, and optionally block AI training crawlers like GPTBot. Output updates live and the page flags the classic empty-Disallow trap. Fully generated in your browser.
How to use
- Fill in the input box above, or click "Fill sample" to try it instantly
- Adjust the options as needed — results update as you type
- Click "Run" to get the result, then copy or export it in one click
Tips
- Runs entirely in your browser — your data never leaves the device
- Completely free, no registration, no usage limits
- Most tools compute in real time as you type