Toolsky

Free Online Tools & Utilities for Every Task.

Robots.txt Generator

Generate Robots.txt Files

Every search engine that visits your site asks for one small file before anything else: robots.txt. Get that file wrong and the consequences run in both directions — private folders get crawled and indexed, or whole sections you want in search results quietly disappear. Writing the file by hand invites exactly those mistakes, because the syntax is unforgiving: one misplaced asterisk or a missing slash changes the meaning of a rule entirely. A robots.txt generator removes the guesswork by turning a few plain choices — which crawlers, which paths, allow or block — into a correctly formatted file you can upload with confidence.

What a Robots.txt File Actually Controls

Robots.txt is a plain-text instruction file that lives in the root of your domain, at yoursite.com/robots.txt, and it follows the Robots Exclusion Protocol, standardised as RFC 9309. When a well-behaved crawler such as Googlebot or Bingbot arrives, it reads this file first and treats its rules as instructions for what it may fetch. Two directives do most of the work. User-agent names the crawler a block of rules applies to — a specific bot, or * for all of them. Disallow lists path prefixes that agent should not crawl, while Allow carves exceptions back out of a broader block. A line such as Sitemap: followed by your sitemap URL tells crawlers where your page list lives; pairing the file with our Sitemap Generator covers both halves of crawl guidance at once.

Two limits are worth knowing before you rely on the file. Robots.txt controls crawling, not indexing with absolute force: a blocked page that other sites link to can still appear in results as a bare URL, because Google may index what it cannot fetch. And it is not a security measure — the file is publicly readable, so listing a sensitive folder there actually advertises its location to anyone curious.

What This Generator Lets You Set

The generator is built around the choices a site owner actually faces, rather than asking you to recall directive syntax:

  • Default behaviour for all crawlers — allow everything, or block everything, as the starting point for the file.
  • Rules for specific bots — add separate blocks for individual crawlers when one search engine or AI bot should be treated differently from the rest.
  • Custom blocked and allowed paths — enter folders such as /admin/, /checkout/ or /search/ and have them formatted as correct Disallow lines.
  • Crawl-delay values — where a bot honours it, you can add a delay between requests to soften the load on a small server; note that Googlebot ignores this directive and uses Search Console settings instead.
  • Sitemap declaration — append your sitemap URL so crawlers find it without a separate submission.

The finished file is shown as clean plain text, ready to copy or download exactly as a crawler expects to receive it.

How to Generate Your Robots.txt in Six Steps

  1. List, on paper first, the areas of your site crawlers should stay out of — admin screens, internal search results, checkout and account pages, staging folders, and bulky parameter URLs.
  2. Choose the default rule for all crawlers: normally "allow all", then block the exceptions.
  3. Add the paths from your list as disallowed entries, keeping the leading slash on each one and remembering that rules match by path prefix.
  4. Add a separate block for any individual bot that needs different treatment, such as a stricter rule for an aggressive scraper.
  5. Enter your sitemap URL so it is written into the file.
  6. Generate the file, copy or download it, and upload it to the root of your domain — not inside a subfolder — then open yoursite.com/robots.txt in a browser to confirm it loads.

Where a Generated Robots.txt Pays Off

Consider a small online shop. The owner wants products indexed but sees no value in crawlers burning budget on the filter pages that spin out hundreds of near-identical URLs, or on /cart/ and /checkout/. The generated file for that shop might look like this:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Allow: /

Sitemap: https://example-shop.com/sitemap.xml

Every product and category page stays crawlable, the wasteful corners are fenced off, and the sitemap line points bots straight at the pages that matter. The same pattern suits a WordPress site fencing off /wp-admin/ (while allowing /wp-admin/admin-ajax.php, which some themes need), or a staging subdomain blocked entirely with a single Disallow: / under a wildcard agent while it is under construction.

Robots.txt Mistakes That Cost You Traffic

The classic accident is leaving a site-wide block in place after a redesign: a Disallow: / written for the staging phase goes live and the site's crawl traffic falls off a cliff within days. Check the file after every migration. Second, remember that blocking a page in robots.txt does not remove it from the index — for removal you need a noindex tag or password protection, and a blocked page cannot be noindexed effectively because the crawler never fetches it to see the tag. Third, syntax slips matter: directives are case-sensitive in path matching, wildcards behave differently across engines, and Google's own RFC 9309 specification defines the formal rules if you ever need the ground truth. Finally, never list genuinely sensitive directories here hoping secrecy will protect them — protect them with authentication instead. After uploading, run the page through our SEO Analyzer to confirm the rest of your on-page setup is equally tidy, and if duplicate URLs are your real problem, the Canonical Generator tackles that from the other side.

Frequently Asked Questions About Robots.txt Files

Where exactly must robots.txt be placed?

In the root directory of the host it governs, so it loads at yoursite.com/robots.txt. A file inside a subfolder is ignored, and each subdomain needs its own file.

Does blocking a page in robots.txt remove it from Google?

Not reliably. It stops crawling, but a page with external links can still be indexed as a URL-only result. For dependable removal, use a noindex meta tag or require a password.

What is the difference between Disallow: / and Disallow: (empty)?

Disallow: / blocks the entire site. An empty Disallow value blocks nothing and means everything may be crawled — the two are opposites separated by one character, which is why generating beats typing.

Can I block one specific search engine only?

Yes. Give that bot its own block starting with its User-agent name and its own rules; crawlers follow the most specific block that names them.

Do all bots obey robots.txt?

Major search engines do, and the protocol is an honoured convention, but nothing technically forces compliance. Rogue scrapers may ignore it entirely, which is why it must never be your only protection.

How often should I update my robots.txt?

Whenever your site structure changes — new sections, a redesigned checkout, a removed staging area — and after every platform migration. A quick review twice a year catches most drift.

Does a crawl-delay help my server?

Some crawlers (Bing, Yandex) honour it and will space out requests. Googlebot ignores crawl-delay entirely; control its rate in Google Search Console instead.

A Small File That Guards the Whole Site

Robots.txt is twenty lines of text at most, yet it decides which parts of your site search engines ever see. Generating it instead of typing it removes the syntax errors that cause the expensive failures, and leaves you a file you can re-check in seconds whenever the site changes.