How to Use Robots.txt in URL Audits

Robots.txt tells automated clients which parts of a website they are allowed or not allowed to request. It can also contain XML sitemap declarations. During URL audits, robots.txt can affect both sitemap discovery and scanning.

URL Verifier Pro supports robots.txt in two main ways: it can check robots.txt for sitemap declarations during sitemap discovery, and it can optionally respect robots.txt rules during scans.

Sitemap discovery and robots.txt

When you enter a root domain or website URL, URL Verifier Pro can check robots.txt for Sitemap: entries. If no sitemap declaration is found, it can also try common sitemap locations such as /sitemap.xml.

This makes it easier to find sitemap URLs without manually opening robots.txt in a browser.

Robots-aware scanning

If robots.txt rule checking is enabled in scan settings, URL Verifier Pro fetches robots.txt for each host and applies the relevant allow/disallow rules. URLs that are disallowed are skipped and marked as Not Allowed instead of being requested.

This is useful when you want an audit to respect a site's published crawl rules.

When to enable robots.txt checks

Enable robots.txt checks when you want to:

• Respect crawl rules during third-party or client site audits.

• Identify important URLs that are blocked by robots.txt.

• Check sitemap URLs against robots rules.

• Avoid requesting URLs a site has explicitly disallowed.

When you might disable robots.txt checks

You may choose to disable robots.txt checks for:

• Internal QA on a website you own or manage.

• Staging environments where robots.txt blocks everything by design.

• Controlled testing where you have permission to check the URL regardless of robots rules.

How to report robots issues

A focused robots report should usually include:

• Original URL

• Robots status

• Status code

• Final URL

• Error details

• Notes

• Tags

Use tags such as robots blocked, review robots or expected block to make follow-up easier.