Skip to main content
Utilnivo

SEO & Webmaster

Utilnivo

Robots.txt Tester

Test whether a URL is allowed in robots.txt.

No registration
  • On your device

Browse more in SEO & Webmaster or all tools.

Private on your device

Your information stays on your device and is not uploaded.

Related tools

Paste robots.txt rules or fetch /robots.txt from a site, then test whether a URL path is allowed for Googlebot, Bingbot, or all crawlers. This tester checks existing rules—use Robots.txt Generator when you need to create a new robots.txt file.

How to use this tool

1. Paste robots.txt or fetch your live file. 2. Enter the URL path to test (for example /admin/). 3. Pick a user-agent. 4. Run the test. 5. Read allowed vs blocked and which rule matched.

Worked example

Example 1: Disallow: /cart blocks /cart/checkout for Googlebot—confirm you intended that. Example 2: Allow: /wp-admin/admin-ajax.php with a broader Disallow: /wp-admin/ — the more specific Allow can keep AJAX reachable while blocking the rest.

When to use this

  • Before launching a new Disallow rule.
  • Debugging why Googlebot skipped a path.
  • Comparing Googlebot vs Bingbot groups.
  • Verifying Allow exceptions under a broad Disallow.
  • After migrating WordPress or Shopify robots rules.

Common examples

  • Disallow: /search blocks faceted search URLs for Googlebot.
  • Allow admin-ajax while Disallow /wp-admin/ for WordPress.
  • Different rules for Googlebot vs Bingbot user-agent groups.
  • Trailing-slash path differences: /blog vs /blog/ matching.
  • Comment lines and blank lines ignored during evaluation.

Common mistakes

  • Using robots.txt instead of authentication for private pages.
  • Blocking CSS/JS needed for rendering.
  • Forgetting sitemap directives live elsewhere.
  • Testing the wrong user-agent group.
  • Assuming robots.txt guarantees deindexing.

How it works

Evaluates a path against robots.txt rules for a chosen user-agent (Googlebot, Bingbot, or *). Longest matching Allow/Disallow wins per common crawler behavior; this is a client-side check, not a live crawl.

Limitations

Robots parsing follows common crawler rules but may not match every search engine implementation exactly.

Privacy and file handling

Your data stays on your device and is not uploaded.

Formula or method

Rules are grouped by User-agent. Matching uses prefix rules; the most specific matching Allow/Disallow typically decides access for that agent.

FAQ

Frequently asked questions

Is this a robots.txt validator?

Yes. It checks whether a path is allowed or blocked under your robots.txt rules for a chosen crawler—useful as a robots.txt checker before you publish changes.

Does a Disallow remove a page from Google?

No. robots.txt can block crawling, but URLs may still appear in results without snippets. Use noindex (while allowing crawl) when you need removal from the index.

Which user-agent should I test?

Test Googlebot for Google Search, and also your general * group if you rely on catch-all rules. Different agents can have different Allow/Disallow sets.

Robots.txt tester vs Google Search Console?

This tool evaluates rules you paste or fetch in the browser. Search Console’s robots tester and URL Inspection reflect Google’s live crawl view for your verified property.

Can robots.txt protect private pages?

No. It is a public hint for well-behaved crawlers. Use authentication and proper access control for private content.

Part of these workflows

This tool is one step in a longer job. Jump straight to your step or open the full workflow guide.

Related pages

Page last reviewed: