robots.txt generator
Build a correct robots.txt — and understand which part of it can hurt you.
Free · No sign-up · Runs in your browser
robots.txt is a small file with a lot of power. One line in the wrong place takes a site out of Google — with no error anywhere, just rankings draining away over the following weeks.
Ready-made rules
Your robots.txt
Save this as robots.txt in your site root — it must load at your-domain.com/robots.txt
Common misconception
Disallow does not remove a page from Google. It stops crawling — the URL can stay indexed with no snippet, and Google will never see a noindex tag on it. To remove a page, use noindex and leave it crawlable.
How to use it
- Pick a preset matching what your site runs on.
- Add your own disallows if there are folders you do not want crawled.
- Enter your sitemap.xml URL so Google finds it immediately.
- Save the file in your site root and check it loads at /robots.txt.
What to know
The most common misconception: Disallow does not remove a page from Google. It stops crawling — the URL can stay indexed with no snippet, and Google will never see a noindex tag on it. To remove a page you use noindex and leave it crawlable.
Tell us briefly what you need. We reply within one working day — with concrete ideas, a timeline and a price.