Skip to main content
Utilnivo

SEO & Webmaster topic guide

Sitemap and robots.txt

Build XML sitemaps and robots.txt files, then validate syntax before submitting to search engines.

4 free tools in this guide — pick one below to get started.

Tools in this guide(4)

Open a tool to calculate, convert, or edit — no signup required.

Related focused guides

Focused pages with tailored examples and FAQs—each reuses the same tool with task-specific guidance.

Overview

Sitemaps help crawlers discover URLs; robots.txt tells crawlers what they may request. Utilnivo’s generators produce starter files you can adapt to your site structure, while validators catch syntax errors before upload.

Use Sitemap Generator when you have a list of URLs and need a compliant XML file. Robots.txt Generator drafts allow/disallow rules and sitemap references. Run Sitemap Validator and Robots.txt Tester on the finished files or live URLs to confirm parsers accept them.

Robots.txt is not a security mechanism—it only guides well-behaved bots. Block sensitive paths with authentication, not disallow rules alone.

How to choose

  • New XML sitemap → Sitemap Generator.
  • Check sitemap XML → Sitemap Validator.
  • Draft crawl rules → Robots.txt Generator.
  • Test live robots.txt → Robots.txt Tester.

Frequently asked questions

Where do I upload sitemap.xml?

Host it on your domain (for example /sitemap.xml) and reference it in robots.txt and Search Console.

Can I block CSS and JavaScript in robots.txt?

You can, but blocking assets can hurt rendering and SEO. Modern guidance is to allow CSS and JS.

Does robots.txt guarantee noindex?

No. Use noindex meta tags or HTTP headers for pages that must not appear in search results.

Step-by-step workflows (6)

Multi-step guides that chain tools from this topic—follow numbered steps instead of guessing tool order.

View all workflows

Related topic guides