URL Verifier Pro Guide

How to Reduce 403 and 429 Errors During Bulk URL Checks

  • URL Verifier Pro

  • Technical guide

URL Verifier Pro guide workspace

When you check URLs in bulk, some websites may block, slow or rate limit requests. The most common signs are 403 Forbidden, 429 Too Many Requests, timeouts, block pages, challenge pages or repeated failed rows.

These responses do not always mean the URLs are broken. They often mean the site is protecting itself from automated traffic.

URL Verifier Pro includes scan controls such as concurrency, per-host concurrency, delay, jitter, retries and backoff to help you scan more responsibly.

What 403 usually means

A 403 Forbidden response means the server refused the request.

Common causes include:

  • Bot protection.

  • Firewall rules.

  • Missing authentication.

  • Blocked User-Agent.

  • Blocked IP address or proxy.

  • Geographic restrictions.

  • Permissions or server configuration.

  • WAF or CDN security rules.

A single 403 may be page-specific. Many 403 responses across one host usually indicate an access rule or blocking pattern.

What 429 usually means

A 429 Too Many Requests response means the server is rate limiting requests.

Common causes include:

  • Too many requests in a short period.

  • Too many requests to the same host.

  • Repeated retries.

  • Aggressive concurrency settings.

  • Proxy networks with poor reputation.

  • Security tools reacting to automated checks.

If you see 429, slow down before retrying the whole list.

Start by reducing scan pressure

The first fix is usually not a proxy or a different User-Agent. It is reducing request pressure.

Try:

  • Lower overall concurrency.

  • Lower per-host concurrency.

  • Add a fixed delay.

  • Add random jitter.

  • Increase backoff after 403 or 429.

  • Use fewer retries.

  • Increase timeout for slow servers.

  • Scan a smaller sample first.

This is especially important for small sites, shared hosting, staging environments and client sites without strong infrastructure.

Use per-host concurrency carefully

Overall concurrency controls the total number of active requests. Per-host concurrency controls how many requests hit the same host at once.

For large URL lists on one domain, per-host concurrency matters a lot.

If you scan 10,000 URLs from one host with high per-host concurrency, you are sending concentrated traffic to that server. Lowering per-host concurrency often reduces blocks without making the audit unusably slow.

Add delay and jitter

A fixed delay spaces requests out.

Jitter adds randomness so requests do not arrive in a perfectly mechanical pattern.

Use delay and jitter when:

  • The site is fragile.

  • The site starts returning 429.

  • The site is on shared hosting.

  • The scan targets one host heavily.

  • You want a friendlier audit profile.

Use backoff for blocked hosts

Backoff pauses or slows requests after blocked or rate-limited responses.

This helps avoid hammering a host that is already refusing requests. It also makes the scan more realistic because it gives the server time to recover or lift temporary rate limits.

Check User-Agent settings

If a site blocks the default client but works in a browser, a browser-style User-Agent may help diagnose the issue. Use this carefully and document it in your results.

Read When to Use a Custom User-Agent for URL Status Checks before changing User-Agent settings across a large audit.

Check authentication and cookies

If protected URLs return 403, the issue may not be rate limiting. It may be missing authentication.

For staging and protected URLs, check:

  • Basic Authentication.

  • Cookie header.

  • VPN access.

  • IP allowlisting.

  • Proxy access.

  • Correct hostname.

See How to Scan Staging or Password-Protected URLs for a safer workflow.

Be careful with proxies

Proxies can help in some approved workflows, but they can also increase 403, 429 and timeout rates. Many websites block known proxy networks.

Before using proxies, compare a small direct scan with a small proxy scan.

If proxy results are worse, do not treat proxy failures as website failures. Read How to Use Proxies for Bulk URL Checking for more detail.

How to report blocked results

Do not mix blocked results with confirmed broken URLs.

Use tags such as:

  • blocked

  • rate limited

  • needs auth

  • retry slower

  • proxy issue

  • user agent review

  • manual check

A useful report should separate:

  • Confirmed 404 or 410 URLs.

  • Server errors.

  • Blocked requests.

  • Rate-limited requests.

  • Proxy failures.

  • Authentication issues.

Suggested safe retry workflow

  1. Filter 403, 429, Failed and Timeout rows.

  2. Save the session.

  3. Lower per-host concurrency.

  4. Add delay and jitter.

  5. Increase backoff.

  6. Rescan only the affected rows.

  7. Compare results.

  8. Tag rows that still fail.

  9. Manually verify important URLs before escalating.

Final advice

The best way to reduce 403 and 429 errors is to treat the target site with care. Start smaller, scan slower, document settings and separate blocked results from true broken URLs. This produces a more trustworthy audit and avoids creating unnecessary load.

Back to guides