Robots.txt Generator
Answer a few quick questions to generate a ready-to-upload robots.txt file for your website.
Your robots.txt
What is a robots.txt file?
robots.txt is a plain text file placed at the root of your domain (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not crawl. It does not guarantee pages stay out of the index, but it controls crawl behaviour and saves crawl budget.
How to use the Robots.txt Generator
- Choose whether crawlers should be allowed everywhere by default, or blocked everywhere.
- List any specific folders or URL patterns you want to block, one per line, such as /wp-admin/ or /cart/.
- Add your XML sitemap URL so crawlers can find it easily.
- Click Generate, copy the output, and upload it as robots.txt to your site root.
Common mistakes to avoid
Do not block CSS or JS files needed to render your pages, and never use robots.txt as your only method to hide sensitive content — use noindex meta tags or authentication for that instead.
Frequently Asked Questions
Where should I upload my robots.txt file?
It must be placed at the root of your domain, for example https://example.com/robots.txt, not inside a subfolder.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. Blocked pages can sometimes still appear in search results without a description if they are linked from elsewhere. Use a noindex meta tag for pages you want fully removed.
Can I have multiple sitemaps in one robots.txt?
Yes, you can add multiple Sitemap: lines, one for each sitemap file or sitemap index.
Is robots.txt required for every website?
It is not strictly required, but having one gives you explicit control over crawler behaviour and is considered SEO best practice.
Need expert help instead of just a tool?
BrightFolio's team can handle your SEO, ads, and content end-to-end.
Get a Free Strategy Call