Make sure AI's bots can actually reach your site

Every crawler hit is checked against the IP ranges the operators publish, so a request calling itself GPTBot only counts as GPTBot when it arrives from an address OpenAI actually publishes. Spoofed traffic is separated out instead of being quietly included. That matters because anyone can set a user agent, and counting the spoofed hits would make a site that is blocking real AI crawlers look perfectly well crawled.

An emerging convention for a plain-text file that tells AI systems what your site contains and where the canonical version of each thing lives, in the spirit of robots.txt but aimed at readers rather than indexers. It is not a standard any engine is obliged to honour, and we do not claim it guarantees anything. Our audit checks both robots.txt and llms.txt, and can generate a starting file from your site structure for you to review.

The crawlability audit names the specific rule or misconfiguration doing it: a robots.txt directive, a firewall or CDN rule, a bot-management setting, or a login wall on a page you meant to be public, along with the change to make. Blocks are frequently unintentional, arriving with a security product rather than a decision, which is why this is the first thing worth ruling out when visibility falls without an obvious cause.

The major AI and AI-adjacent crawlers, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, so the bots behind ChatGPT, Claude, Gemini and Perplexity are covered. Crawler tracking and the crawlability audit are included on every plan, unlike engine coverage, because whether a bot can reach your site is a prerequisite for everything else rather than a premium report. New crawlers are added as operators publish and document their ranges.

Not every visit means the same thing. Hits are sorted by why the bot came: a live agent fetching a page to answer somebody's question right now, a search-index crawl building a retrievable corpus, or a training crawl. The distinction is practical. Live-agent traffic on a page is direct evidence it is being used to answer questions, while a training crawl tells you far less about today's answers and is the tier many teams choose to restrict.

That is a business decision and we deliberately do not make it for you. Blocking training crawlers while allowing live-agent and search-index crawlers is a common middle position: it keeps your content out of a training corpus while leaving you citable in answers. Blocking everything is a coherent choice if you are protecting content, but expect to disappear from AI answers as a result. This tooling lets you set the policy and then verify it is behaving as intended.