URL Verifier Pro Guide

When to Use a Custom User-Agent for URL Status Checks

  • URL Verifier Pro

  • Technical guide

URL Verifier Pro guide workspace

A User-Agent is a request header that identifies the client making the request. Browsers send one. Search engine bots send one. URL checking tools send one too.

Most of the time, the default User-Agent in URL Verifier Pro is enough. There are cases, though, where changing the User-Agent can help you understand why a site responds differently to different clients.

This guide explains when to use a custom User-Agent, when not to use one and how to keep the results useful.

If you are seeing different results between Chrome and a checker, start with Why a URL Works in Your Browser but Fails in a URL Checker.

Why User-Agent matters

Some websites change behaviour based on the User-Agent.

They may:

  • Block unknown clients.

  • Serve different redirects.

  • Return different content types.

  • Require a browser-like User-Agent.

  • Treat search bot User-Agents differently.

  • Route mobile and desktop requests differently.

  • Trigger bot protection for automation tools.

This means the same URL can return different results depending on the request headers.

When to use the default User-Agent

Use the default User-Agent when you want a neutral technical check.

It is usually the right choice for:

  • General URL status audits.

  • Sitemap health checks.

  • Redirect chain checks.

  • Internal QA.

  • Most migration verification.

  • Large URL lists where you do not need browser-specific behaviour.

The default User-Agent gives you a consistent baseline. That is important when you want repeatable results.

When to try a browser User-Agent

A browser User-Agent can be useful when a site blocks unfamiliar clients or behaves differently for real browsers.

Try a browser-style User-Agent when:

  • Many URLs return 403.

  • A site works in Chrome but fails in the scan.

  • A CDN or firewall appears to block automated clients.

  • A staging environment expects browser traffic.

  • You are checking how redirects behave for normal users.

Do not use this to disguise abusive scanning. Use it for legitimate debugging on sites you own, manage or have permission to test.

When to use a search bot User-Agent

Search bot User-Agent presets can help investigate whether a site is showing different responses to crawlers.

Use this carefully.

Search bot checks can be useful for:

  • Comparing normal visitor responses with crawler responses.

  • Checking whether important pages redirect differently for bots.

  • Investigating suspected bot-specific blocks.

  • Reviewing robots and crawl access issues.

Do not claim to be a search engine bot unless you have a legitimate reason to test that behaviour. Also remember that some sites verify search bot identity by IP, so changing the User-Agent alone may not reproduce true crawler access.

Recommended workflow

  1. Run a baseline scan with the default User-Agent.

  2. Save the session or export the results.

  3. Change the User-Agent setting.

  4. Rescan a small sample of affected URLs first.

  5. Compare status codes, final URLs, redirect chains and errors.

  6. If the change explains the issue, rescan the relevant rows.

  7. Document the User-Agent used in your notes or report.

Do not immediately rescan 100,000 URLs with a new User-Agent. Test a sample first.

What to compare

When testing User-Agents, compare:

  • Status code

  • Final URL

  • Redirect chain

  • Response time

  • Content type

  • Server header

  • Cache-Control header

  • Error details

  • Block-page indicators

  • Robots status

If only the User-Agent changes and the response changes too, that is useful evidence.

Common patterns

Default User-Agent gets 403, browser User-Agent gets 200

The server may be blocking unknown automated clients. Review whether this is intentional.

Browser User-Agent redirects differently

The site may use device, region or client detection. Check whether those redirects are expected.

Search bot User-Agent gets blocked

The site or CDN may be blocking crawler-like traffic. Check firewall and bot protection settings.

User-Agent makes no difference

The problem is likely elsewhere. Check authentication, cookies, proxies, robots.txt, TLS, DNS, rate limits or server errors.

Reporting User-Agent findings

When User-Agent affects results, document it clearly.

A useful note might say:

Default User-Agent returned 403. Browser User-Agent returned 200. Review bot protection rules.

Include columns such as:

  • Original URL

  • Status code

  • Final URL

  • Redirect chain

  • User-Agent used

  • Error details

  • Notes

  • Tags

Avoid common mistakes

Do not change User-Agent settings randomly during one audit. Results become hard to compare.

Do not use a search bot User-Agent and present the result as if it were a normal user result.

Do not rely on User-Agent changes alone to bypass rate limits or access controls.

If you see many 403 or 429 responses, also read How to Reduce 403 and 429 Errors During Bulk URL Checks.

Final advice

Treat User-Agent settings as a diagnostic tool. Start with a clean baseline, change one thing at a time and document what changed. That makes your audit more reliable and much easier to defend later.

Back to guides