BOSTIC / FREE TOOLS
Robots.txt checker
Inspect your robots.txt and test a path for Googlebot, Bingbot or all crawlers. Follow the rule behind the result.
No account requiredNo AI credits
Up to 64 KB and 2,000 lines.
Your results will appear here
Start with a URL. We’ll show what we found and what it means.
Public URL checks use Bostic servers and may reuse a result for 5 minutes. Do not submit private or signed URLs.
Know the scope
From a question to a next step.
A rules preview for common Allow/Disallow patterns. Robots.txt controls crawling, not guaranteed indexing. Advanced encoding and crawler-specific behavior may differ.
User-agent: * Disallow: /private/ Path: /private/prices Rules preview: blocked from crawling, not proof of removal from search.
Illustrative example, not a check of your website. Run the tool above for your own result.
BOSTIC.AI
Turn a page check into a fix list.
Create a project, verify your website, then review saved audit findings and affected URLs. A single-page check does not describe your whole site.