Crawled 1.17 Million Times, Four Pages Indexed. Google Explains What Went Wrong.

A site owner posted a striking pair of screenshots: Googlebot had crawled their site 1.17 million times in a few weeks, and Search Console showed four indexed URLs out of roughly 180,000 pages. The homepage was not among them.

John Mueller’s answer is worth reading as a general lesson rather than as a diagnosis of one unlucky site. He identified two problems.

Problem one: a technical failure that made indexing pointless

“A bunch of the URLs that are indexed from your site, they all redirect to the homepage, so it seems like there are still some big technical issues that you need to resolve.”

Read that carefully. The pages Google had indexed were redirecting somewhere else. Indexing is downstream of a whole chain — discovery, crawling, rendering, canonicalisation — and a redirect chain that collapses everything onto the homepage means the pages Google chose to keep were the ones that led nowhere distinct.

This is the failure mode that heavy crawling hides. Crawl volume looks like attention. It is not. Google crawled 1.17 million times because there were 180,000 URLs to look at and something kept making it look again — not because the site was valuable.

Problem two: programmatic pages at a scale nobody can maintain

Mueller’s second point was about the site’s category. He described it as a large product aggregator, and said that building and maintaining one well is not trivial: you end up with an enormous number of URLs, apparently with programmatic auto-generated content, and convincing search engines it is worthwhile — and users that it is valuable — will be hard.

Then the line that should be printed above every “we’ll generate 10,000 pages” plan: Google is generally not a fan of programmatic SEO.

Why this matters beyond aggregate sites

The temptation to scale pages programmatically is not limited to product aggregators. It shows up as comparison pages for every competitor, landing pages for every city, listing pages for every category permutation, and directories of every tool in a niche.

Every one of those produces URLs. The question Google keeps asking is whether the resulting pages are worth existing individually, and the answer is judged page by page, not site by site — which is why one section of a large site can rank while another is discarded entirely.

Two tests are worth applying before generating anything at scale:

If the fourth answer is yes, you have found pages that should not exist separately.

The off-site version of the same question

The same logic applies, inverted, to the pages other people publish about you. A directory with 180,000 auto-generated listings has the aggregate problem: it may be crawled heavily and valued lightly, and a citation on one of its pages inherits that condition.

This is why the useful filter for submission targets is not domain authority but whether the page you land on was composed by someone. A listing on a curated directory that reviews submissions, and on https://backlinksubmit.org the platforms are drawn from a handpicked list rather than a scraped one, is a different kind of asset from an entry in a generated index — even when both are technically live and both pass a link check.

What to do if you recognise your own site here

Start with the redirects. If indexed URLs are collapsing onto your homepage, nothing else you do matters until that is fixed. Then take the largest cluster of programmatic pages and ask the four questions above; the honest result is usually that a fifth of the URLs should exist and the rest should be consolidated.

The instinct will be to fix it by adding pages. The fix is almost always removing them.

FAQ

Does heavy crawling mean Google likes my site?

No. Crawling is discovery, not endorsement. A site with hundreds of thousands of URLs can be crawled constantly and indexed almost not at all.

Does Google penalise programmatic SEO?

Not as a policy. Google’s stated position is closer to indifference that tips into distrust: generated pages are judged individually, and most of them do not pass.

What’s the fastest diagnostic?

Check whether your indexed URLs redirect somewhere else. If a large share collapse onto the homepage, you have a technical problem to solve before a content problem.

How do I know which of my pages should exist?

Ask whether a person would be annoyed if the page did not exist, and whether it could be merged into a sibling without losing anything. If it can be merged, it should be.