Free tool

AI crawler robots.txt checker

Direct answer: Paste a robots.txt (or try a URL) to see whether major search and AI search crawlers appear allowed at the site root. Allow Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, and Claude-SearchBot if you want search/AI answers to use your pages. Training bots (GPTBot, ClaudeBot, Google-Extended) are a separate policy choice.

Site or robots.txt URL Or paste robots.txt

Check

Why this matters

Blocking a search crawler can stop Indexed → Found → Mentioned before content work starts. Google’s AI features still depend on normal Search eligibility — not on llms.txt.

How long indexing takes → · Is llms.txt worth it? →

← Back to home · Start with one sentence