What the robots.txt file is
robots.txt is a plain-text file that lives at the root of your domain (https://your-site.com/robots.txt) and tells search engine crawlers which parts of your site they may or may not visit. It is the very first thing Googlebot checks before indexing your pages, so a broken file can hide your entire site from Google or, the other way around, leave a private dashboard exposed.
This generator builds the file for you from simple options: you pick which bots you target, which paths you block with Disallow, which exceptions you permit with Allow, an optional Crawl-delay, and your sitemap URL. Everything runs in your browser; nothing is sent to a server. Copy the result and upload it to your hosting root.
How to use the generator
- Click a preset to start from a base: Allow everything, Block everything, or Block /admin and /private.
- In each block, type the
User-agent(use*for every bot, or a name likeGooglebot). - Add one path per line under Disallow (what gets blocked) and Allow (the exceptions).
- If you need it, set a
Crawl-delayin seconds and your sitemap URL. - Copy the generated text and save it as
robots.txtat the root of your domain.
What each directive means
| Directive | What it does |
|---|---|
User-agent | Names the robot the block’s rules apply to. * means every bot. |
Disallow | A path the bot must not crawl. An empty Disallow: blocks nothing. |
Allow | An exception that permits a path inside a blocked folder. |
Crawl-delay | Seconds to wait between requests. Google ignores it; Bing and Yandex honor it. |
Sitemap | Absolute URL of your sitemap so engines can find your pages. |
Paths are matched by prefix from the root: Disallow: /admin/ blocks /admin/, /admin/users, and everything below it. The most specific rule wins, which is why an Allow can rescue one subfolder inside a broader Disallow.
Worked example
Imagine a shop that wants to hide its admin panel and cart while keeping everything else visible, and to declare its sitemap. You set up a single block with User-agent: *, two Disallow lines (/admin/ and /carrito/), and the sitemap URL. The generator outputs:
User-agent: *
Disallow: /admin/
Disallow: /carrito/
Sitemap: https://asistente-rd.com/sitemap.xmlThat file asks every bot to skip /admin/ and /carrito/, allows the rest of the site, and points crawlers to the sitemap. Upload it to https://asistente-rd.com/robots.txt and verify it by opening that URL in your browser.
Frequently asked questions
Does robots.txt keep a page out of Google?
Not reliably. Disallow prevents crawling, but if other sites link to the URL, Google may still index it without its content. To guarantee a page stays out of results, use a meta name="robots" content="noindex" tag in the page’s HTML (and do not block it in robots.txt, or Google will never see the noindex).
Where must the file live?
Always at the domain root: https://your-site.com/robots.txt. Crawlers only look there. A robots.txt placed inside a subfolder (/blog/robots.txt) has no effect at all.
Is an empty Disallow: the same as Disallow: /?
They are opposites. A Disallow: with no path means “block nothing”, i.e. crawl everything. Disallow: / blocks the entire site. Confusing the two is a classic mistake that de-indexes a whole website.
Does Google respect Crawl-delay?
No. Googlebot ignores the Crawl-delay directive; you tune its rate through Google Search Console instead. Bing, Yandex, and others do read it, so it is still worth including if server load is a concern.
Can I have several User-agent blocks?
Yes. You can create a * block with general rules and specific blocks for individual bots such as Googlebot or Bingbot. Each crawler applies the single block whose User-agent best matches its name, not the combination of all of them.