robots.txt Generator
Generate robots.txt files to control how search engines crawl your site
Rule 1
Additional Settings
Generated robots.txt
What is robots.txt?
A robots.txt file tells search engine crawlers which URLs they can access on your site. This is used mainly to avoid overloading your site with requests.
Where to Place
The robots.txt file must be placed in the root directory of your website. For example: https://example.com/robots.txt
Important Notes
- robots.txt is advisory, not enforced
- Blocked pages can still be indexed
- Use noindex meta tag to prevent indexing