SEO & Webmaster topic guide
Sitemap and robots.txt
Build XML sitemaps and robots.txt files, then validate syntax before submitting to search engines.
Topic guides
Browse focused guides that compare related tools and link to the best starting point for your task.
- Meta tags and previewsGenerate and preview title tags, meta descriptions, Open Graph, and social cards before you publish a page.
- Sitemap and robots.txtBuild XML sitemaps and robots.txt files, then validate syntax before submitting to search engines.
- Schema and structured dataGenerate JSON-LD schema markup and validate structured data before publishing rich results.
- Technical SEO auditCheck redirects, canonical tags, broken links, headings, and HTTP status before or after a site launch.
- Crawl, headers, and performanceInspect HTTP headers, security headers, keyword density, and Core Web Vitals concepts for healthier pages.
- Campaign tracking and URLsBuild UTM campaign links and clean URL slugs for analytics-ready marketing pages.
Tools in this guide
Overview
Sitemaps help crawlers discover URLs; robots.txt tells crawlers what they may request. Utilnivo’s generators produce starter files you can adapt to your site structure, while validators catch syntax errors before upload.
Use Sitemap Generator when you have a list of URLs and need a compliant XML file. Robots.txt Generator drafts allow/disallow rules and sitemap references. Run Sitemap Validator and Robots.txt Tester on the finished files or live URLs to confirm parsers accept them.
Robots.txt is not a security mechanism—it only guides well-behaved bots. Block sensitive paths with authentication, not disallow rules alone.
How to choose
- New XML sitemap → Sitemap Generator.
- Check sitemap XML → Sitemap Validator.
- Draft crawl rules → Robots.txt Generator.
- Test live robots.txt → Robots.txt Tester.
Frequently asked questions
Where do I upload sitemap.xml?
Host it on your domain (for example /sitemap.xml) and reference it in robots.txt and Search Console.
Can I block CSS and JavaScript in robots.txt?
You can, but blocking assets can hurt rendering and SEO. Modern guidance is to allow CSS and JS.
Does robots.txt guarantee noindex?
No. Use noindex meta tags or HTTP headers for pages that must not appear in search results.
Related workflows
These multi-step guides chain tools from this category for common jobs.