URL Verifier Pro Guide

How to Find Noindex and Nofollow Pages in Bulk

  • URL Verifier Pro

  • Technical guide

URL Verifier Pro guide workspace

Noindex and nofollow signals can be useful when they are intentional. They can also cause serious problems when they are left on important pages after a migration, staging launch, CMS update or template change.

A bulk URL checker that extracts robots directives helps you find these signals across a large URL list instead of checking page source one URL at a time.

URL Verifier Pro can report robots meta signals, noindex and nofollow flags, and related response information during a URL audit.

What noindex means

A noindex directive tells search engines not to index a page. It may be found in:

  • A meta robots tag in the HTML.

  • An X-Robots-Tag response header.

  • Template-level SEO settings.

  • CMS or plugin settings.

Noindex is often intentional for search results pages, internal pages, filtered URLs, account pages and staging environments. It is a problem when it appears on pages that should rank.

What nofollow means

A nofollow directive tells search engines not to follow links on the page, or not to treat links in the usual way. It may be used intentionally, but it should be reviewed if it appears across important templates.

During audits, nofollow is most useful as a signal to investigate rather than an automatic error.

When to run a noindex and nofollow audit

Run this check when:

  • A site has just launched.

  • A staging site has gone live.

  • Organic traffic dropped after a release.

  • A CMS or SEO plugin was updated.

  • XML sitemaps contain unexpected pages.

  • Important pages are not being indexed.

  • A template change affected many page types.

  • You are reviewing robots directives before migration sign-off.

Recommended workflow

  1. Import or discover the URLs you want to check.

  2. Run the scan with robots and metadata columns visible.

  3. Filter for rows where Noindex is Yes.

  4. Filter for rows where Nofollow is Yes.

  5. Review status code, final URL and canonical URL before deciding whether the signal is a problem.

  6. Tag rows as expected noindex, unexpected noindex, nofollow review or needs SEO review.

  7. Export a focused report.

For robots.txt rules specifically, see How to Use Robots.txt in URL Audits.

What to review first

Noindex pages in XML sitemaps

If a URL is in an XML sitemap and also has noindex, review it. Sitemaps should generally contain URLs that you want indexed.

Noindex on important landing pages

High-value commercial pages, category pages, content hubs and product pages should usually not be noindex unless there is a specific reason.

Noindex after migration

This is common when staging settings are moved to production. Check launch-critical templates first.

Nofollow on templates

If nofollow appears on every page using a template, check whether the directive is intentional.

Conflicts with canonical tags

A page that is noindex and canonicalises elsewhere should be reviewed carefully. The intended indexing signal should be clear.

Useful columns

For a noindex and nofollow audit, include:

  • Original URL

  • Final URL

  • Status code

  • Redirect chain

  • Robots meta tag

  • Noindex flag

  • Nofollow flag

  • X-Robots-Tag header where available

  • Canonical URL

  • Page title

  • Content type

  • Error details

  • Notes

  • Tags

Suggested tags

Use tags that separate expected from unexpected behaviour:

  • unexpected noindex

  • expected noindex

  • nofollow review

  • sitemap noindex

  • staging setting

  • template issue

  • needs SEO review

  • fixed

How to report this properly

Do not export every noindex row as an error. Some noindex pages are meant to be noindex.

A useful report should separate:

  • Expected noindex pages.

  • Unexpected noindex pages.

  • Sitemap URLs with noindex.

  • Important pages with noindex.

  • Pages where noindex conflicts with canonical or hreflang signals.

  • Rows that need manual SEO review.

Important limitation

URL Verifier Pro reports the directives it can detect from the response it receives. If directives are injected only after JavaScript runs, they may not be visible in the initial HTML response.

That is still valuable because it shows what a direct HTTP request receives from the server, which is often what you need for technical URL audits.

Final check

After fixes are made, rescan the affected URLs and filter again for Noindex and Nofollow. Export only the remaining rows that need review. This makes the follow-up report much easier to act on.

Back to guides