FREE TOOL
AI crawlability checker: can AI crawlers read your site?
A free AI crawlability check. AfterLaunch reads the rules your robots.txt sets for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the other AI crawlers, loads your homepage, scores what the search crawlers can reach out of 100 and names the top fix.
Free, no signupAllow Perplexity Search in robots.txt.
Example result
The 2 checks behind the score
Your robots.txt, worth 40 points. Does it let in the 4 search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot and Googlebot?
Your homepage, worth 30 points. Does it load for a plain crawler, without a noindex tag?
The score scales those 70 points to 100. A check the run could not read is left out, so it never counts as a fail.
What you get back
A score out of 100, and the top fix if there is one.
Both checks in full, with the change to make for each, free and with no email.
Most fixes are 1 line: an Allow rule in robots.txt, or a noindex tag removed.
Which AI crawlers the check covers
Robots.txt names crawlers by their user-agent token. The check reads the rules your file sets for 10 of them, and the 4 search crawlers decide the 40 crawler points.
OAI-SearchBot (OpenAI Search) fetches pages for ChatGPT search, Claude-SearchBot (Claude Search) for Claude, PerplexityBot (Perplexity Search) for Perplexity and Googlebot (Google Search) for Google.
GPTBot (OpenAI training) and ClaudeBot (Claude training) collect pages to train models. CCBot (Common Crawl) builds an open archive of the web that many models are trained on.
Google-Extended (Google AI controls) is a token rather than a separate crawler: it tells Google whether your pages may be used for its AI models. ChatGPT-User (ChatGPT user requests) fetches a page when someone asks ChatGPT to open it, and Bingbot (Bing) crawls for Bing.
None of those 6 changes the score, because the score is about the four crawlers that build the search indexes for ChatGPT, Claude, Perplexity and Google.
How to fix a blocked crawler
Open your robots.txt and find the group that blocks the crawler: a User-agent line with its token, followed by Disallow: /. Change the rule to Allow: /, or give the crawler a group of its own, with User-agent: OAI-SearchBot on one line and Allow: / on the next.
A crawler follows the group that names it and skips the User-agent: * group, so a group of its own is enough even when everything else is blocked.
If your homepage carries a robots meta tag with noindex in it, remove the tag and run the check again.
The check fetches your llms.txt too but leaves it out of the score. The llms.txt checker, linked below, reads that file in full, and Findable checks your structured data along with the rest of how AI search reads your site.
Common questions
Is my site crawlable by AI?
Run the check above. It reads your robots.txt for the 4 search crawlers behind ChatGPT, Claude, Perplexity and Google, loads your homepage and scores what they can reach out of 100. A full score means every check the run could read passed.
What is an AI crawlability test?
A test of whether the crawlers AI search engines use can get to your pages. This one checks 2 things: whether your robots.txt allows the search crawlers, and whether your homepage loads for a plain crawler without a noindex tag.
Does a high score mean AI answers will name me?
No. It means the crawlers can get in. Whether answers name you depends on what your pages say, and what other sites say about you. The free scan shows whether ChatGPT and Perplexity name you today.
Is it safe to run on any site?
Yes. It makes 3 plain requests and changes nothing. You can run it on a competitor too.
The organic growth engine for AI search and SEO
AfterLaunch helps you show up everywhere people search.
Free deep scan, no card.