Robots.txt Generator
Create a robots.txt file that tells search engines and AI crawlers what they may crawl.
Your robots.txt appears below as you choose options.
How to use the Robots.txt Generator
- Choose what search engines may crawl, and list any folders to block.
- Add your sitemap address.
- Download the file and upload it to the top folder of your site, so it opens at yoursite.com/robots.txt.
What robots.txt can and cannot do
A robots.txt file asks crawlers not to visit certain parts of your site. Reputable search engines and AI companies follow it. It is not a security measure: the file is public, bad bots ignore it, and a blocked page can still appear in search results if other sites link to it. To keep a page out of Google completely, use a noindex tag or a password instead.
Frequently asked questions
Where does the robots.txt file go?
In the root folder of your site, so its address is exactly https://yoursite.com/robots.txt. It does not work in a subfolder.
Does blocking a page remove it from Google?
No. It stops Google reading the page, but the address can still be listed. Use a noindex meta tag to remove a page, and do not block that page in robots.txt, or Google will never see the tag.
Will blocking AI crawlers hurt my Google ranking?
No. The AI crawler names are separate from Googlebot. "Google-Extended" only controls use of your content for Google's AI models, not normal search.
Should I set a crawl delay?
Usually not. Google ignores it, and it only helps if crawlers such as Bingbot are overloading a small server.
Is what I type sent to a server?
No. This tool runs entirely in your browser, so nothing you enter leaves your device.