robots.txt Generator

Free robots.txt Generator

This robots.txt Generator helps you create a correct robots.txt file in seconds — no syntax to memorize. Pick a preset or define your own rules, and the tool builds a clean, valid file you can copy or download and upload to your site's root. A well-formed robots.txt tells search-engine crawlers which parts of your site they may or may not request, helping you manage crawl budget and keep low-value pages out of the way.

What Is robots.txt?

robots.txt is a plain-text file placed in the root of your domain (for example https://example.com/robots.txt) that follows the Robots Exclusion Protocol. When a compliant crawler such as Googlebot visits your site, it reads this file first to learn which URLs it is allowed to crawl. It is a crawling directive, not a security mechanism — it asks well-behaved bots to stay out of certain paths.

robots.txt Syntax

• User-agent — The crawler a rule block applies to. Use * to target all bots, or a specific name like Googlebot.
• Disallow — A path crawlers should not request, e.g. Disallow: /admin/. An empty Disallow: means "allow everything."
• Allow — Explicitly permits a path, useful for carving an exception inside a disallowed folder (e.g. allow /wp-admin/admin-ajax.php while blocking the rest of /wp-admin/).
• Crawl-delay — Requests a pause (in seconds) between hits. Honored by some engines like Bing; Google ignores it.
• Sitemap — The full URL of your XML sitemap, which helps engines discover all your pages.

Built-in Presets

• Allow everything — A permissive file that lets all crawlers access the whole site.
• Block everything — Disallows all crawlers from the entire site (handy for staging or development environments).
• WordPress — A sensible default that blocks /wp-admin/ and a few low-value paths while keeping admin-ajax.php reachable.

How to Use the Generator

1. Choose a preset, or set the User-agent and add your own Disallow / Allow paths (one per line).
2. Optionally set a Crawl-delay and your Sitemap URL.
3. The robots.txt output updates live as you type.
4. Click Copy or Download to save the file.

Where to Upload robots.txt

The file must live at the very root of your domain — https://yoursite.com/robots.txt — not in a subfolder. Upload it via FTP, your hosting file manager, or (on WordPress) an SEO plugin that manages a virtual robots.txt. After uploading, confirm it loads in a browser and validate it in Google Search Console's robots.txt report.

Common Mistakes to Avoid

• Do not use robots.txt to hide pages from search results. Blocking a URL stops crawling but the page can still be indexed if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
• Never block your CSS and JavaScript. Google needs them to render and understand your pages.
• Avoid Disallow: / on a live site unless you truly want to block everything — it can deindex your entire domain.
• Remember it is case-sensitive and paths are matched as prefixes from the root.

robots.txt vs. the Meta Robots Tag

They solve different problems. robots.txt controls crawling (whether a bot may request a URL). The meta robots tag and the X-Robots-Tag header control indexing (whether a crawled page may appear in results). For reliable removal from search, use noindex — and make sure the page is not blocked in robots.txt, or crawlers will never see the noindex directive.

Frequently Asked Questions

Do I need a robots.txt file?

Not strictly — if you want everything crawled, you can omit it. But having one lets you point crawlers to your sitemap and exclude low-value paths.

Will robots.txt stop bad bots or scrapers?

No. Only well-behaved crawlers obey it. Malicious bots ignore the file entirely, so it is not a security tool.

Can I list more than one sitemap?

Yes. You can add multiple Sitemap: lines; this generator includes the one you enter, and you can paste extra lines manually if needed.

Is this tool free and private?

Yes. The file is built entirely in your browser — nothing is uploaded — and there are no limits or sign-up.