FREE TOOL
Robots.txt checker: what may crawlers read?
A free robots.txt checker. AfterLaunch reads your file, flags a site-wide block and the lines crawlers skip, and shows what Googlebot, Bingbot and the AI crawlers may read.
Free, no signupStarting from nothing? The robots.txt generator writes the file.
What the robots.txt checker reads
The file at /robots.txt on your site, read once with a plain request, the way a crawler reads it.
It says whether the file answers as text, whether its rules block every crawler from the whole site, and whether it names your sitemap.
It lists lines a crawler skips: misspelt directives, rules written before any User-agent line, and noindex, which Google stopped obeying in robots.txt on 1 September 2019.
Which crawlers it checks
The search crawlers: Googlebot, Bingbot, OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. A block on any of these keeps your pages out of that search.
The AI training and control tokens: GPTBot, ClaudeBot, Google-Extended and CCBot. These are shown and never marked as faults, because blocking them is a choice about training and does not stop AI search reading your site.
How to fix a blocked crawler
Find the group that blocks it: a User-agent line naming it, or User-agent: *, followed by Disallow: /. Give the crawler a group of its own with Allow: /, or remove the rule.
A crawler follows the group that names it and skips the * group, so a group of its own is enough even when everything else stays blocked.
Common questions
What if my site has no robots.txt?
Crawlers read a missing file as permission to read everything. Nothing is blocked, but there is nowhere to name your sitemap, so the check suggests publishing one.
Does a robots.txt keep a page out of Google?
Not reliably. It stops crawlers reading the page, but a blocked page can still be listed if other sites link to it. A noindex tag on the page is what keeps it out.
Is it safe to run on any site?
Yes. It makes one plain request for /robots.txt and changes nothing.
Keep going
The organic growth engine for AI search and SEO
AfterLaunch helps you show up everywhere people search.
Free deep scan, no card.