Free toolsrobots.txt checker
Free robots.txt Checker
Paste a domain and see its robots.txt read the way an analyst would: which AI crawlers and search bots it blocks, the sitemaps it declares, and what that means in plain English.
What robots.txt actually controls
Every crawler that visits a site is supposed to read example.com/robots.txt first. That file is a set of rules keyed to user-agents — the name a crawler identifies itself with — that say which paths it may or may not fetch. It is advisory: reputable crawlers obey it, but it is not a lock. A single misplaced Disallow: / can quietly wall a whole site off from the bots you actually want.
Why the AI-crawler lines matter now
The classic question was “can Googlebot and Bingbot reach my pages?” The newer one is whether you want your content feeding AI answers and training sets. Those crawlers each announce themselves with a distinct user-agent:
- GPTBot — OpenAI's crawler for training data.
- ClaudeBot — Anthropic's crawler.
- PerplexityBot — Perplexity's answer engine.
- Google-Extended — Google's opt-out token for Gemini and AI training, separate from Googlebot.
Blocking Googlebot and allowing GPTBot — or the reverse — is a real decision, and most sites make it by accident. This tool calls out the AI crawlers first, next to the search bots, so you can see the posture at a glance instead of parsing the file by hand.
Questions
Is my robots.txt a security control?
No. It only asks crawlers not to fetch a path; anyone can still read it, and the file itself is public. Use authentication for anything that must stay private.
Should I block AI crawlers?
It depends on your goal. If you want visibility in AI answers, allow them. If your content is your product and you do not want it in training sets, disallow GPTBot, ClaudeBot, PerplexityBot and Google-Extended specifically — blocking Googlebot does not cover them.
Does “Disallow” remove a page from Google?
No. It stops the fetch, but a disallowed URL can still be indexed from external links. To keep a page out of the index, let it be crawled and add a noindex directive.
robots.txt is one line of the read.
A blocked crawler is a symptom. Itto is an AI SEO agent: give it a domain and it crawls the site, pulls live search data and answers like an analyst — cited, caveated, with the next question ready. The robots.txt check is the free front door.
Any domain · one Google sign-in · no Analytics or Search Console access.