FREE TOOL
Robots.txt generator, with the AI crawlers by name
Choose what search engines and AI crawlers may read: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot for search, GPTBot, ClaudeBot, Google-Extended and CCBot for training. Add the paths to keep private and your sitemap, then copy or download the file.
Free, no signup# Every other crawler
User-agent: *
Allow: /
Publish it at the root of your domain, so it loads at /robots.txt. Then run the robots.txt checker on your site.
Already have a robots.txt? The robots.txt checker tests it.
What the generator writes
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming crawlers, followed by the Allow and Disallow rules those crawlers should follow.
The default rule becomes the User-agent: * group, which every crawler without a group of its own reads. Allow everything writes Allow: /, and Block everything writes Disallow: /.
A crawler you set differently from the default gets a group of its own. A crawler obeys the one group that names it and skips the * group, so its own group is enough even when everything else is blocked.
Paths you list become Disallow lines in every group that allows crawling, and the sitemap address becomes a Sitemap line at the end. Crawlers that read the file find your sitemap from it.
The crawlers, by token
Googlebot crawls for Google Search and Bingbot for Bing. OAI-SearchBot fetches pages for ChatGPT search, Claude-SearchBot for search in Claude, and PerplexityBot for Perplexity. Blocking one of these keeps your pages out of what that search reads.
GPTBot collects pages that OpenAI may use to train its models, and ClaudeBot does the same for Anthropic. CCBot builds Common Crawl, an open archive of the web that many models are trained on.
Google-Extended is not a separate crawler. It is a token Google reads to decide whether pages Googlebot has already fetched may be used for its AI models. Blocking it does not change how Google Search crawls you.
To be found in AI search without being used for training, allow the search crawlers and block the training ones.
Where the file goes
Save it as plain text named robots.txt at the root of your domain, so it loads at yoursite.com/robots.txt. A file anywhere else is ignored, and each subdomain needs its own.
Well-behaved crawlers follow robots.txt, but nothing forces them to, so a page that must stay private needs a login. A Disallow line only asks.
Once it is live, run the robots.txt checker to read it back the way a crawler does.
Common questions
Should I block GPTBot and ClaudeBot?
That is your call. Blocking them keeps your pages out of future training data for OpenAI and Anthropic models. It does not affect OAI-SearchBot or Claude-SearchBot, which fetch pages for search, so you can block training and still be read by ChatGPT search and Claude.
Does blocking Google-Extended remove me from Google?
No. Google-Extended only tells Google whether your pages may be used for its AI models. Googlebot still crawls for Google Search under its own rules.
Will a Disallow line take a page out of search results?
Not reliably. Disallow stops a crawler reading the page, but a search engine can still list the address if other sites link to it. To keep a page out of results, let crawlers read it and add a noindex robots meta tag.
Is anything I enter sent to AfterLaunch?
No. The file is written in your browser as you change the form, and nothing is stored or sent.
Keep going
The organic growth engine for AI search and SEO
AfterLaunch helps you show up everywhere people search.
Free deep scan, no card.