Robots.txt Generator
Generate a properly formatted robots.txt file in seconds. Control how search engines and AI bots crawl your website.
Access Control
Paths to Block
AI Bot Rules
Enable this to prevent AI bots from scraping your content for training data.
Sitemap URL
Adding a sitemap helps search engines discover and index all your pages.
Crawl Delay (optional)
Sets delay between crawler requests. Only use if your server has limited resources. Google does not honor this directive.
# robots.txt generated by google4seo.com # Generated: 2026-03-24 User-agent: * Allow: / Disallow: /admin Disallow: /api
How to Use Your robots.txt
- 1Download or copy the generated file
- 2Place it in the root directory of your website
- 3Verify it's accessible at yourdomain.com/robots.txt
- 4Test in Google Search Console under Settings > robots.txt
What is robots.txt?
The robots.txt file is a simple text file placed in your website's root directory that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol, a standard used by all major search engines including Google, Bing, and Yahoo.
While robots.txt can help manage crawl budget and prevent indexing of non-essential pages, it's important to note that it is a directive, not an enforcement mechanism. Well-behaved bots will respect the rules, but malicious crawlers may ignore them. For true access control, use server-side authentication.
Do use robots.txt to:
- Manage crawl budget for large sites
- Prevent indexing of admin areas
- Block duplicate content pages
- Point crawlers to your sitemap
Don't use robots.txt to:
- Hide sensitive data (use auth instead)
- Block individual pages from Google (use noindex)
- Prevent link juice flow (use nofollow)
- Replace proper access controls