Skip to main content

SEO & Webmaster

Robots.txt Tester

Test whether a URL is allowed in robots.txt.

No registration
  • On your device

Browse more in SEO & Webmaster or all tools.

How do I check if Googlebot can crawl /admin/?

Paste your robots.txt, enter the path /admin/, and choose Googlebot as the user-agent. A Disallow: /admin rule typically blocks /admin and /admin/login; a more specific Allow can override a broader Disallow for the matching path. This is a client-side rule check—not a live crawl of your site—so confirm important paths after deploys.

Private on your device

Your information stays on your device and is not uploaded.

Related tools

Paste robots.txt rules or fetch /robots.txt from a site, then test whether a URL path is allowed for Googlebot, Bingbot, or all crawlers. This tester checks existing rules—use Robots.txt Generator when you need to create a new robots.txt file.

How to use this tool

1. Paste robots.txt or fetch your live file. 2. Enter the URL path to test (for example /admin/). 3. Pick a user-agent. 4. Run the test. 5. Read allowed vs blocked and which rule matched.

Worked example

Disallow: /cart blocks /cart/checkout for Googlebot. Allow: /wp-admin/admin-ajax.php with Disallow: /wp-admin/ can keep AJAX reachable while blocking the rest of /wp-admin/.

When to use this

  • Confirming /admin or /cart rules after a CMS robots change.
  • Checking Googlebot vs Bingbot User-agent groups separately.
  • Validating Allow overrides inside a broader Disallow.
  • Debugging why a path still appears blocked in Search Console.
  • Spot-checking staging robots before production promote.

Common examples

  • Disallow: /admin blocks /admin/settings for Googlebot.
  • Allow specific AJAX path while Disallowing /wp-admin/.
  • Bingbot vs Googlebot different User-agent groups.
  • Trailing slash path mismatch vs rule prefix.
  • Confirm /cart still crawlable after a CMS robots change.

Common mistakes

  • Assuming a path test equals a live crawl of the whole site.
  • Testing with * when a bot-specific group is more specific.
  • Forgetting trailing-slash prefix differences.
  • Editing robots live without testing Disallow on money pages.
  • Confusing robots rules with meta robots noindex on HTML.

How it works

Evaluates a URL path against robots.txt Allow/Disallow rules for a chosen user-agent. Longest matching rule wins per common crawler behavior—this is a client-side check, not a live site crawl.

Limitations

Robots parsing follows common crawler rules but may not match every search engine implementation exactly.

Privacy and file handling

Your data stays on your device and is not uploaded.

Formula or method

Rules are grouped by User-agent. Matching uses prefix rules; the most specific matching Allow/Disallow typically decides access for that agent.

FAQ

Frequently asked questions

Is this a robots.txt validator?

Yes. It checks whether a path is allowed or blocked under your robots.txt rules for a chosen crawler—useful as a robots.txt checker before you publish changes.

Does a Disallow remove a page from Google?

No. robots.txt can block crawling, but URLs may still appear in results without snippets. Use noindex (while allowing crawl) when you need removal from the index.

Which user-agent should I test?

Test Googlebot for Google Search, and also your general * group if you rely on catch-all rules. Different agents can have different Allow/Disallow sets.

Robots.txt tester vs Google Search Console?

This tool evaluates rules you paste or fetch in the browser. Search Console’s robots tester and URL Inspection reflect Google’s live crawl view for your verified property.

Can robots.txt protect private pages?

No. It is a public hint for well-behaved crawlers. Use authentication and proper access control for private content.

Does this crawl my whole site?

No. It evaluates one path against the robots.txt rules you paste or fetch for a chosen user-agent. Use it to verify Allow/Disallow intent, then confirm important URLs in Search Console.

Part of these workflows

This tool is one step in a longer job. Jump straight to your step or open the full workflow guide.

Related pages

Page last reviewed: