Crawl rules
Generated robots.txt
\U0001F4A1 Place the finished file at the root of your domain: https://example.com/robots.txt
Create a valid robots.txt file with allow and disallow rules, crawl-delay and your sitemap — using ready-made presets or fully custom directives. Copy or download in one click.
Control exactly which pages search engines may crawl.
Start from Allow all, Block all or a WordPress preset, or choose Custom to build rules from scratch. Presets fill the fields instantly.
Use * to target every crawler, or name a specific bot like Googlebot to write rules that apply only to it.
Click "Add rule" to block or permit folders and pages, for example /admin/ or /private/. Each rule appears in the output as you type.
Drop in your sitemap URL, set an optional crawl-delay, then Copy or Download the file and upload it to your site root.
Everything you need to know about the Robots.txt Generator.
A robots.txt file is a plain text file at the root of your website that tells search engine crawlers which pages or folders they may or may not access. It is the first file most bots check, and it helps you keep private or duplicate areas out of search results.
Choose a preset or select Custom, set the user-agent, then add Disallow rules for paths you want to block and Allow rules for exceptions. Add your sitemap URL and an optional crawl-delay. The file builds live on the right — copy it or download it and upload it to your domain root.
It must live in the root directory of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers will not find it in a subfolder. Upload it via FTP, your hosting file manager, or your CMS.
Not reliably. Robots.txt controls crawling, not indexing. A blocked URL can still appear in search if other sites link to it. To keep a page out of the index, use a meta robots "noindex" tag (you can create one with our Meta Tag Generator) instead of, or alongside, robots.txt.
Crawl-delay asks crawlers to wait a set number of seconds between requests, which can reduce server load. Google ignores it, but some other engines respect it. Only add it if your server struggles under crawl traffic; otherwise leave it blank.
Keep optimizing \u2014 explore the rest of the SEO toolkit.