URL Verifier Pro Guide

How to Scan Staging or Password-Protected URLs

  • URL Verifier Pro

  • Technical guide

URL Verifier Pro guide workspace

Staging sites, QA environments and protected pages often behave differently from public websites. They may require Basic Authentication, cookies, allowlisted IPs, VPN access, custom headers, special User-Agents or proxy access.

URL Verifier Pro can help check protected URLs when the site accepts the authentication method you configure. It supports HTTP Basic Authentication, Cookie headers, custom User-Agent settings and proxy options.

Use these features only for websites and environments you are authorised to test.

Common protected URL scenarios

You may need this workflow for:

  • Staging websites behind Basic Authentication.

  • Client preview sites that require a login cookie.

  • Internal QA environments.

  • Pre-launch migration checks.

  • Protected development domains.

  • Sites only accessible through a corporate proxy.

  • Environments that block unknown User-Agents.

  • Pages that require a session cookie to return the correct response.

If a URL works in your browser but fails in the app, read Why a URL Works in Your Browser but Fails in a URL Checker.

Start with access first

Before scanning, confirm that your machine can access the environment.

Check:

  • VPN is connected if required.

  • DNS resolves the staging host.

  • Firewall rules allow access.

  • Basic Authentication credentials are correct.

  • The cookie is current and scoped to the right site.

  • The proxy route works if required.

  • The URL list uses the correct staging domain.

If the site is not reachable outside your browser, URL Verifier Pro may not be able to reach it either.

Basic Authentication workflow

Use Basic Authentication when the site shows a browser username and password prompt before loading.

Recommended steps:

  1. Open settings.

  2. Enable Basic Authentication.

  3. Enter the username and password.

  4. Run a small test scan first.

  5. Confirm protected URLs return the expected status.

  6. Scan the wider list once the sample works.

Watch for 401 and 403 responses. A 401 usually means authentication is missing or rejected. A 403 usually means the request was understood but access was refused.

Cookie header workflow

Use a Cookie header only when you understand what cookie is required and have permission to use it.

Recommended steps:

  1. Sign in to the site in your browser using a test account.

  2. Identify the required cookie value.

  3. Add the Cookie header in URL Verifier Pro settings.

  4. Test a small number of URLs.

  5. Confirm the content and status codes match what you expect.

  6. Avoid exporting or logging sensitive cookie values.

Cookies can expire. If a scan suddenly starts returning login pages or redirects, refresh the cookie.

Custom User-Agent workflow

Some staging sites or security tools respond differently to unknown clients.

Use a custom User-Agent when:

  • The site blocks the default request.

  • The staging environment expects browser-like traffic.

  • You are comparing browser-style and default responses.

  • The client has asked you to use a specific User-Agent.

Read When to Use a Custom User-Agent for URL Status Checks before using this as a diagnostic method.

Proxy workflow

Some protected environments are only reachable through a proxy or corporate route.

Recommended steps:

  1. Test the proxy with a small URL set.

  2. Confirm the proxy supports the target protocol.

  3. Check credentials and formatting.

  4. Compare a direct scan with a proxy scan where possible.

  5. Tag rows where proxy behaviour affects results.

See How to Use Proxies for Bulk URL Checking for a fuller workflow.

Scan settings for staging

Staging environments are often slower and more fragile than production.

Use conservative settings:

  • Lower overall concurrency.

  • Lower per-host concurrency.

  • Longer timeout if the environment is slow.

  • Modest retry settings.

  • Request delay if the environment is fragile.

  • Jitter if multiple requests hit the same host.

  • Backoff if the server starts blocking.

Do not scan a fragile staging server like a production CDN.

Robots.txt on staging

Many staging sites block all crawlers with robots.txt. That may be intentional.

If robots.txt checking is enabled, URL Verifier Pro may mark staging URLs as Not Allowed. If you own or manage the environment and need to test the responses anyway, you may choose to disable robots.txt checking for that scan.

Document that decision in your audit notes.

Privacy and logging

Protected scans can involve sensitive URLs, cookies or credentials.

Before scanning:

  • Use test accounts where possible.

  • Do not use personal login sessions unless necessary.

  • Keep scan logging disabled unless troubleshooting.

  • Avoid exporting sensitive cookies or internal URLs unnecessarily.

  • Share reports only with authorised people.

  • Remove credentials from any support request unless asked.

Useful columns

For staging and protected URL checks, include:

  • Original URL

  • Final URL

  • Status code

  • Status text

  • Redirect chain

  • Content type

  • Response time

  • Error details

  • Server header

  • Cache-Control header

  • Notes

  • Tags

Final advice

Protected URL scanning is mostly about controlling the test environment. Start with a small sample, confirm authentication works, keep settings gentle and document exactly what access method was used. That makes the results far more useful for developers, SEOs and QA teams.

Back to guides