What it does
robots.txt is the plain text file at the root of your site that tells crawlers which paths they may fetch. AI companies now run several crawlers each, and they don't all do the same job.
This generator sorts them by purpose: bots that fetch a page because someone asked an assistant a question, bots that build an AI search index, and bots that collect training data. You choose for each one, and for the main search engines, and get a file you can copy.
How to use it
- 1Pick a starting point under “Start from”: “Allow all AI”, “Block training only” or “Block all AI”.
- 2Adjust single crawlers with their “Allow” and “Block” switches. They're grouped under AI assistants answering a question now, AI search indexes, and AI training data, with the company next to each name.
- 3Check the “Search engines” section. Googlebot, Bingbot, DuckDuckBot, Applebot and YandexBot are allowed unless you change them.
- 4Add your “Sitemap URL (optional)”, press “Copy”, and upload the result as /robots.txt at the root of your domain.
Blocking training without leaving AI answers
Most sites that want to limit AI use start with “Block training only”. That blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider and CCBot, and keeps the answer and search bots allowed.
This matters because the answer and search bots are how assistants like ChatGPT, Claude and Perplexity find your pages and link to them. Block those and you can drop out of their answers.
What Google-Extended and Applebot-Extended do
These two aren't separate crawlers. Googlebot and Applebot still do the fetching. The Extended tokens tell Google and Apple whether they may use what they fetched for their AI models.
Blocking them doesn't remove you from Google Search or Apple's search features. To leave search, you'd block Googlebot or Applebot instead, which most sites don't want.
What robots.txt can't do
robots.txt is a request. Well-behaved crawlers follow it; others can ignore it. It also doesn't hide a page that's already public, and blocking a URL doesn't always keep it out of search results if other sites link to it. Use a noindex tag for that.
Questions
Will blocking GPTBot remove my site from ChatGPT?
Not by itself. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when someone asks. Each has its own switch here.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Google's AI models. Search crawling and ranking go through Googlebot.
How fast do crawlers notice a new robots.txt?
It depends on the crawler. Google usually caches robots.txt for up to a day. Others re-read it on their own schedule.
Do I need a separate file for each subdomain?
Yes. Each host reads its own robots.txt, so blog.example.com and example.com each need one at their root.
How do I check what my current file allows?
Use the AI crawler access report to see every crawler's status at a glance, or the robots.txt tester to check one path for one crawler.