Robots.txt Generator

Your robots.txt tells crawlers which parts of your site they may visit. Choose the paths to keep private and decide which AI crawlers may use your content.

Runs entirely in your browser. Nothing you type is sent anywhere.

One path per line, for example /admin/ or /cart

Block AI crawlers
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Frequently asked questions

Does blocking AI crawlers affect my Google rankings?
No. Google-Extended only controls use of your content for Gemini; Googlebot, which ranks your pages, is separate. Blocking GPTBot or ClaudeBot does not change Google rankings.
Does robots.txt keep a page out of Google?
Not reliably. A blocked page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag and let crawlers reach it.

Go further with GlobalSEO

Keyword research, rank tracking, site audits, backlinks, AI visibility and Search Console insights in one place.

Start free