robots.txt Generator

Robots txt generator: create a valid robots.txt file fast

This robots txt generator builds a correct robots.txt file for your site without making you memorize the syntax. Choose which crawlers to allow or block, list the paths to keep out of search, add your sitemap, and copy the finished file to your site root at /robots.txt. It runs free in your browser with no signup, and the output works anywhere — a custom site, WordPress, or a hosted store. One stray character in a hand-typed file can hide your whole site from Google, and this tool keeps the syntax valid so that does not happen by accident.

A robots.txt file is a plain-text list of instructions telling search engine crawlers which parts of your site they may request. It is the first file Googlebot looks for. Done well, it steers crawl budget toward your important pages; done badly, it can quietly remove you from search.

How to use the robots.txt generator

1. Set a default rule — allow all crawlers, or restrict them.
2. Add any specific user-agents you want to treat differently, such as blocking a scraper while allowing Googlebot.
3. List the directories or files to Disallow, like /admin/ or /cart/.
4. Add your Sitemap: URL so crawlers can find your full page list.
5. Copy the generated file and upload it to your site root so it lives at https://yoursite.com/robots.txt.

That last step matters: a robots.txt file only works at the root. A free robots.txt generator saves you the syntax, but you still place the file yourself.

How robots.txt directives work

The file is made of groups. Each group names a User-agent (the crawler) and then lists rules for it. Disallow blocks a path, Allow re-permits a path inside a blocked area, and Sitemap points to your XML sitemap. Paths match as prefixes, and * is a wildcard. The key thing to understand: robots.txt controls crawling, not indexing. To keep a page out of search results entirely you need a noindex meta tag, not a disallow rule.

robots.txt directive reference

Common robots.txt examples

These ready-made patterns cover most needs. Use them as a starting point and adjust the paths to your site.

robots.txt for WordPress

WordPress serves a virtual robots.txt by default, but it is minimal. A sensible robots txt for WordPress blocks the admin area while keeping admin-ajax.php open — themes and plugins need it — and always lists your sitemap. Do not block /wp-content/ wholesale, because that can stop Google from loading the CSS and images it needs to render your pages. This WordPress robots.txt generator produces exactly that pattern when you pick the WordPress preset, so you get a safe file without trial and error.

Crawling versus indexing: the distinction that trips people up

The most common robots.txt misunderstanding is treating it as a way to keep pages out of Google. It is not. robots.txt governs crawling — whether a bot is allowed to fetch a URL. It does not govern indexing — whether a URL appears in search results. Those are separate steps, and the gap between them causes real trouble.

Here is the catch. If you Disallow a page, Google will not crawl it, but if other sites link to that page, Google can still list the URL in results — often with no description, because it was never allowed to read the page. Worse, because it cannot crawl the page, it never sees a noindex tag you may have added there. So the reliable way to remove a page from search is the opposite of what people expect: allow crawling and add a noindex meta tag (or an X-Robots-Tag header) to the page itself. Use Disallow to manage crawl budget and keep bots out of low-value areas, not as a privacy or removal tool.

Mistakes to avoid

• Accidentally blocking the whole site. A stray Disallow: / under User-agent: * hides everything. Double-check before uploading.
• Using robots.txt to hide private data. The file is public, and every path you list is visible to anyone. Use real authentication for sensitive areas.
• Expecting Disallow to remove a page from Google. It blocks crawling, not indexing. A blocked URL can still appear in results; use noindex to remove it.
• Forgetting the sitemap line. Adding Sitemap: helps crawlers discover every page faster.
• Putting the file in the wrong place. It must sit at the domain root; a robots.txt in a subfolder is ignored.
• Blocking CSS and JS. If Google cannot fetch your styles and scripts, it cannot see your page as users do, which can hurt rankings.

How crawlers read the file

When a bot arrives at your domain it requests /robots.txt before anything else, reads the groups, and applies the rules for its own user-agent. If a bot finds a group that names it specifically, it uses that group and ignores the catch-all User-agent: * group entirely — so a Googlebot-specific group must repeat any rules you also want Googlebot to follow. Within a group, the most specific matching rule wins, which is how an Allow can carve an exception out of a broader Disallow. Lines the bot does not understand, like Crawl-delay for Google, are simply skipped rather than causing an error.

A few formatting details keep the file valid. Paths are case-sensitive and must start with a slash. Each directive sits on its own line. Comments begin with # and are ignored. The file should be served as plain text from the root over the same protocol as your site, and it should be small — there is a size limit, and a bloated file with hundreds of rules is both harder to reason about and slower to apply. When in doubt, fewer, clearer rules beat a long list nobody can audit.

Tips and related tools

Keep the file short and deliberate — every line should have a reason. After publishing, verify it with a robots txt checker such as the tester in Google Search Console, and re-check it whenever you redesign. List your sitemap, allow your assets, and block only what genuinely should stay out of search.

Find this and more under our website management tools hub. To inspect a page's raw markup while you work, use the Online HTML Viewer; to identify a site's platform before tuning its crawl rules, try the WordPress Theme Detector. For the social side of crawling, the Open Graph Generator and Twitter Card Generator control how your pages look when shared.

Frequently asked questions

Is this robots.txt generator free? Yes. It is a free robots.txt generator with no signup. Build, copy, and reuse as many files as you like.

Where do I put the robots.txt file? Upload it to your site root so it is reachable at https://yoursite.com/robots.txt. It will not work from a subfolder.

How do I make a robots.txt for WordPress? Choose the WordPress preset. It blocks /wp-admin/, keeps admin-ajax.php open, and adds your sitemap line.

How do I block all crawlers? Use User-agent: * followed by Disallow: /. That is the robots txt disallow all pattern — use it carefully, since it hides your whole site.

Does Disallow remove a page from Google? No. It only blocks crawling. To keep a page out of search results, use a noindex meta tag instead.

Should I add my sitemap? Yes. A Sitemap: line helps search engines discover all your pages more efficiently.

Can I block one specific bot? Yes. Add a group with that bot's User-agent and a Disallow: / rule while leaving the default group permissive.

How do I check my robots.txt is correct? Use a robots txt checker such as the tester in Google Search Console, which shows whether a given URL is allowed or blocked.