Check public crawler policy
Parse robots.txt for major AI search, training, classic search, common crawlers, and Content-Signal declarations at the exact URL path.
AI crawler access check
Check AI Crawler Access Signals for Your Website
Test GPTBot, OAI-SearchBot, OAI-AdsBot, ClaudeBot, PerplexityBot, Google-Extended, robots.txt, Cloudflare Content-Signal including use levels, Applebot, Applebot-Extended, headers, meta robots, sitemap, llms.txt, and fetcher context in one transparent report.
What it checks
More than a robots.txt glance
A crawler policy can look open while a page still blocks discovery through headers, meta tags, redirects, thin rendered content, missing AI-readable discovery files, or confusing bot roles.
Inspect page-level signals
Review HTTP status, redirects, meta robots, X-Robots-Tag, canonical, readable text, and JSON-LD.
Find AI-readiness gaps
Check sitemap, llms.txt, llms-full.txt, non-standard fetcher context, and copy-ready fixes without promising guaranteed AI visibility.
Boundaries
A check, not a guarantee
This tool diagnoses public access signals that site owners control. It does not bypass bot defenses, log into websites, or promise citation in any AI answer surface.
Visibility mode
Find accidental blocks that may prevent AI search or retrieval systems from reading public pages.
Protection mode
See which training or retrieval bots you are allowing, then choose a policy intentionally.
Evidence first
Every recommendation ties back to a visible rule, header, tag, status, missing file, or documented bot behavior.
Tool matrix
One scanner, focused readiness reports
Start with the full AI crawler access scan, then use the focused pages when the job is llms.txt validation, technical AEO readiness, or AI search visibility prerequisites.
AI Crawler ListReview documented user agents, source links, categories, and robots.txt applicability.llms.txt CheckerValidate llms.txt and llms-full.txt structure alongside sitemap, robots.txt, and page signals.AEO CheckerRun a technical Answer Engine Optimization readiness check with evidence-backed fixes.AI Search Visibility CheckerCheck technical visibility readiness before moving into prompt or citation tracking.Robots.txt AI CheckerInspect AI crawler policy at the exact submitted URL path.
Tool choice
Use the right check for the job
AI crawler access is one layer. Compare it with visibility tracking and llms.txt validation so teams do not treat one passing score or one Content-Signal line as a full AI search strategy.
Robots policy checkers
Best for confirming whether a specific user agent is allowed or blocked at one URL path.
AI visibility platforms
Best after technical blockers are fixed, when teams need prompt sampling, citations, and share-of-answer tracking.
llms.txt validators
Best for checking AI-readable discovery files, but they still need robots.txt, sitemap, metadata, and readable pages around them.
Crawler tokens vs use controls
Best for separating actual crawl access, such as Applebot, from AI-use control tokens, such as Applebot-Extended.
Content-Signal declarations
Best for spotting search, AI input, AI training, and optional immediate/reference/full use preferences published alongside crawler rules.