Robots.txt Generator
Your robots.txt tells crawlers which parts of your site they may visit. Choose the paths to keep private and decide which AI crawlers may use your content.
Runs entirely in your browser. Nothing you type is sent anywhere.
One path per line, for example /admin/ or /cart
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart Disallow: /search User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
Frequently asked questions
- Does blocking AI crawlers affect my Google rankings?
- No. Google-Extended only controls use of your content for Gemini; Googlebot, which ranks your pages, is separate. Blocking GPTBot or ClaudeBot does not change Google rankings.
- Does robots.txt keep a page out of Google?
- Not reliably. A blocked page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag and let crawlers reach it.
Go further with GlobalSEO
Keyword research, rank tracking, site audits, backlinks, AI visibility and Search Console insights in one place.
Start free