SEO topic clusters: Measure them from your crawl, not your keyword list

Every guide to topic clusters starts with a keyword list. A keyword list works when the pages don’t exist yet. On a site with 2,400 published pages, the hard question is a different one: which of those pages already belong together, which have drifted, and do your internal links follow the topics at all?

A keyword list can’t answer that, because it describes search demand, not your site. So on large sites I measure clusters instead of planning them: from the text of every page and from the links editors placed in that text. I call the two approaches blueprint clusters and survey clusters. A blueprint says what you want to build. A survey records what is already standing.

This post is the survey method I run on one B2B software site, with the real parameters, the thresholds, the runs I threw away, and what each test does and doesn’t prove.

Key Takeaways

• 38.7% of English in-content links stay inside a measured cluster, against 3.2–4.0% for shuffled labels – the clusters match how editors already link.
• 0.882 AUC on 444 page pairs linked after I chose the embedding recipe: linked pages sit closer in embedding space than random pairs.
• 33–41% of service pages fall into noise under the most stable settings (ARI 0.95–0.97), so I chose a less stable one that loses 14%.
• 4 actions, one per page – keep, grow, relocate, prune – come from cluster fit plus Search Console data; only a human prunes.
• 3 places where clusters change content creation: link suggestions, overlap warnings in briefs, and merge candidates backed by shared Search Console demand.

What is a topic cluster, and why does it help SEO?

A topic cluster is a pillar page on a broad subject plus the pages that cover its subtopics, joined by internal links in both directions. If you want the standard walk-through, Semrush’s guide to topic clusters and Ahrefs’ 10-minute version cover it well, and Semrush has a separate piece on pillar pages.

The SEO case rests on two mechanics. The first is discovery and context: Google finds pages and understands how they relate through crawlable links with descriptive anchor text (Google Search Central). The second is topical authority, the idea that a site covering a subject thoroughly earns more trust on it. Ahrefs has a good explainer on topical authority. I treat it as a working model, since nobody outside Google can measure it directly.

The ranking pages agree on the definition, and almost all of them describe the same workflow: pick a topic, research keywords, group them, map them to pages, write, interlink.

Blueprint or survey: Which one does your site need?

A blueprint is the right tool when the pages don’t exist yet. A survey is the right tool when they do. Keyword clustering – grouping queries that share ranking URLs, as in Ahrefs’ and Semrush’s guides – is a blueprint method. It tells you what to build and which page should target which queries. It can’t tell you which of 2,400 existing pages belong together, which have wandered off-topic, or whether your links cross topics.

The survey runs in five steps:

1. Crawl – export the page text, the status codes, and every link placed inside body copy.
2. Embed – turn each page’s text into a vector, so pages with similar meaning sit close together.
3. Cluster – group the vectors into topics and name them.
4. Validate – check the clusters against the in-content link graph, which the clustering never saw.
5. Act – give every page a role and one action, and feed the clusters into link suggestions, briefs, and merges.

There are two ways to run it. The light version needs no code: export in-content links from your crawler, label pages with the clusters you think you have, and count how many links stay inside a label. If the share is low, your blueprint and your site have parted ways. The full version – embeddings, UMAP, HDBSCAN, roles, and actions – needs Python and is what the rest of this post documents.

Here is what the survey says about one page. A blog post listing well-known companies that run progressive web apps has a text vector that lands in the Web development cluster, with a similarity of 0.608 to that cluster’s center – an Expansion page, at the edge of its topic. Its similarity to the Mobile development center is 0.644, 0.036 higher. The rules make it a relocate candidate, and an editor decides what that means: re-link it from the mobile pages, or rewrite it as a web development piece.

The setup behind the numbers

Every number in this post comes from one site, so here is exactly what that site and that data are.

• Site – a B2B software company: blog, service pages, and glossary. The current run covers 2,436 URLs, 2,302 of them in English. A smaller second-language section (134 pages) is placed onto the English clusters, never clustered on its own.
• Data window – a weekly Screaming Frog crawl since 28 September 2026. Full run 2 (30 September) is the accepted baseline; run 9 (27 October) is the latest weekly update. The checks re-run for this post use the 27 October crawl.
• Search Console – 30-day clicks, impressions, and average position per page arrive with each crawl; the page actions use that window. Query-level data for merges comes from the Search Analytics API, 12 months back.
• Stack – Screaming Frog SEO Spider for the crawl and the page text export, Python, sentence-transformers for the embeddings, umap-learn, scikit-learn (HDBSCAN, ARI, ROC AUC), SciPy for cluster matching, and Postgres with pgvector on Supabase as the store.
• Limits – one site, one industry, mostly one language. Every threshold below is calibrated on this site. On yours, rerun the tests before you copy the numbers.

What should you exclude before clustering?

Exclude every URL that has no text of its own, and every link the template adds. Both create structure that says nothing about topics.

Archives are the worst offenders. In my first run, author, tag, and pagination archives formed the two largest “clusters” on the site, 331 and 114 pages, with a cohesion of 0.92–0.98. A cohesion that high is suspiciously perfect, and for a simple reason: the crawler extracts zero words from them, so their vectors land on top of each other. Of 1,284 pages that the original brief listed as missing from the content inventory, 753 were archives.

My scoping rules:

1. Scope – keep only the sections your content process links to and from. Here that is the blog, the service pages, and the glossary.
2. Statuses – keep every status code in scope (200, 301, 404, noindex). Only indexable pages get clustered, but later steps need to know a page now redirects before they suggest a link to it.
3. Text – drop pages with no extracted text, and strip any sentence that appears on 20 or more pages before embedding.
4. Links – count only links placed in body copy (Screaming Frog’s link position “Content”). Menus, footers, and breadcrumbs stay out. Rotating “latest posts” tiles get counted separately.

The link rule matters more than it looks. Of the link inventory my content process already held, 72% were navigation links. A site-wide menu connects every page to every other page, which helps crawling and tells you nothing about topics.

How do you turn page text into vectors that respect topics?

Each page gets two vectors from paraphrase-multilingual-MiniLM-L12-v2: one for the header (title, H1, and meta description) and one for the body, the normalized mean of up to 12 chunks of 100 words. I add the two unit vectors with equal weight and normalize the sum. Before chunking, I strip every sentence that repeats on 20 or more pages – newsletter boxes, CTAs, and legal lines, which otherwise pull unrelated pages together. The 12-chunk cap stops a long guide from drowning out its own headline topic.

I chose the recipe by testing it against the site’s own links. For each pair of pages joined by an in-content link, and for the same number of random page pairs, I compute cosine similarity and ask how well that one number separates the two groups. ROC AUC is the probability that a random linked pair scores higher than a random unlinked pair: 0.5 means no signal, 1.0 means perfect separation (scikit-learn). These were the candidates, scored on the 28 September crawl:

The body carries most of the signal, and the header adds a little on top. e5-small was no better and twice as slow.

That table picked the winner and scored it on the same pairs, so it flatters the winner. Re-running it on a later crawl doesn’t fix that, because most links on a site stay put. The clean test is links that didn’t exist yet: 444 English page pairs linked in the 27 October crawl but not on 28 September, so they played no part in the choice. Against 444 random pairs, the stored vectors score 0.882 (95% bootstrap interval 0.86–0.91). The recipe holds on links it has never seen.

A re-run on the full 27 October link graph answered a question I hadn’t checked. MiniLM reads at most 128 tokens, and a 100-word chunk is longer than that 90% of the time (median 145 tokens), so the model never sees roughly the last eighth of each chunk. On 13,395 linked pairs and as many random ones, cutting chunks at 120 tokens instead of 100 words moved the AUC from 0.885 to 0.886. The truncation costs almost nothing here, so I kept 100-word chunks – a pragmatic call for this site’s English pages, not a rule for longer texts or other languages.

Two caveats. The result comes from one site, not a benchmark – a different language mix or page type could favor another model. And the test trusts your editors: on a site where in-content links are sparse or careless, the AUC tells you less.

Which UMAP and HDBSCAN settings produced usable topic clusters?

The setting I use is UMAP with 15 neighbors, reduced to five dimensions (min_dist 0, cosine), then HDBSCAN with min_cluster_size=20, min_samples=3, run on three seeds (0, 1, 2). UMAP compresses each 384-dimension vector while keeping near neighbors near. HDBSCAN then finds clusters as dense regions and leaves the pages in between as noise – pages it declines to assign to any cluster.

Stability is measured with the Adjusted Rand Index (ARI): how closely two clusterings agree, where 1.0 means identical and 0 means chance (scikit-learn). I compare seed 0 with seed 1 and seed 0 with seed 2, on the pages both runs assign to a cluster.

For this post I re-ran the parameter sweep on the current vectors: 2,302 English pages, 12 settings, three seeds each.

The most stable settings lost. Leaving min_samples unset makes it default to min_cluster_size, which pushes ARI to 0.95–0.97 – and drops a third or more of the service pages into noise. Service pages are the commercial pages a cluster audit exists to place, so I traded a few points of repeatability to keep them on the map. Raising min_cluster_size to 40 failed the other way: without min_samples, one cluster swallowed 31% of the pages, and with it the runs stopped agreeing (ARI 0.78). The original sweep on the 28 September crawl ranked the settings the same way.

One more lesson from the sweep: the link test from the next section can’t pick parameters on its own. Coarser maps keep more links inside clusters simply because the clusters are bigger – 30 / 40 / 3 keeps 51% of links in-cluster, with one cluster holding a third of the site. That’s why every full run must also pass two guards before I review it: the largest cluster may hold no more than 15% of pages, and noise may not exceed 35%. Both are my working limits, not published standards.

The accepted production run, full run 2, came out at 40 clusters, 12.9% noise, a largest cluster of 10%, and an ARI of 0.82 (seed 0 vs 1) and 0.89 (seed 0 vs 2). Its numbers differ from the sweep’s because it is a different measurement: an earlier snapshot, plus a second-level split step described below. That’s also why the sweep shows 39 clusters and run 2 has 40. Run 9, the current one, keeps run 2’s 40, because a weekly run never reclusters.

What comes out is a map of 40 named topics. Some of the clusters on this site:

• Artificial intelligence – the largest cluster, 233 English pages, and unstable across seeds: a big, loose topic.
• Web development – 144 English pages and the lowest cohesion on the site (0.662).
• Mobile development – the one large cluster that split cleanly into two: native app development and cross-platform apps.
• Ecommerce development – the commerce area broke into four smaller clusters between run 1 and run 2.
• Cybersecurity – 67 pages.
• SaaS development and MVP development – 20 pages each and among the tightest topics (cohesion 0.845 and 0.797).
• Company news and history – 49 English blog posts and not a single service page.

Names follow one rule: a short phrase naming the topic of the pages, not a list of top terms. One LLM call proposes names for all clusters at once, a script checks each against the naming rules, and an editor’s name is never overwritten.

How do you validate topic clusters with internal links?

The clusters are more credible when an independent signal agrees with them. In the accepted full run, 38.7% of English in-content links join two pages from the same cluster – 5,395 of 13,940 links between English pages in scope. A link that touches a noise page counts as not in-cluster.

The baseline is a permutation, not a formula. I keep every link exactly where it is – same source pages, same target pages, noise pages still noise – and shuffle the cluster labels among the clustered pages. The shuffle keeps the real shape of the link graph and the real cluster sizes, and breaks only the tie between a page and its topic. Over 1,000 shuffles, the in-cluster share averages 3.6%, 95% of shuffles land between 3.2% and 4.0%, and the highest is 4.3%. That range is the spread of chance results, not a confidence interval for the real share – and 38.7% is nowhere near it.

What this proves is narrow, and worth saying precisely. The clusters agree with the way editors have linked the site. It doesn’t prove the topics are semantically right, that Google groups the pages the same way, or that more in-cluster links would lift rankings. ARI can’t fill that gap either: it only shows that repeated runs agree with each other, and a partition can be stable and still group the wrong pages. The links are a better witness than ARI because editors placed them for readers, long before any embedding existed.

I also use the test to choose between full runs:

Run 4 used the same seeds as run 2 and the same stored vectors for every unchanged page; about 22 pages had changed between the crawls. It was no less stable across seeds (ARI 0.92 and 0.85), so stability couldn’t separate the two. The link test could, in a narrower sense than “run 2 is right”: run 2 agrees better with the links editors placed, by this metric. That’s the claim I act on. I can’t say which of the 22 pages moved the result – I didn’t isolate it – but I know UMAP reacts to small input changes: on the same pages, rounding the stored vectors to six decimals, and nothing else, took one run from 41 clusters to 39.

The test only works if humans placed the links. If your body links come from a tag-based related-posts widget, there is no independent signal to test against.

When should you split a cluster further?

Split a cluster only when the sub-split survives a change of seed. Every cluster above 5% of English pages gets clustered again on its own (UMAP 15, HDBSCAN 12/3), and I keep the split only if the mean ARI across seeds is 0.8 or higher – my working cut-off, not a published one.

In run 2, only mobile passed: 182 pages into two clusters, ARI 0.98 and 1.0. Artificial intelligence (0.63 / 0.54), web development (0.28 / 0.52), and commerce (0.75 / 0.81) stayed whole. A forced seven-way commerce split from run 1 did not replicate on other seeds. Sometimes one big, messy cluster is the honest answer, and a pillar page with clear sections serves it better than seven invented subtopics.

Which pages are Core, Focus, Expansion, or Outlier?

A page’s role comes from its cosine similarity to its cluster’s centroid, against two thresholds I froze in calibration run 1: Core at 0.766 or above, Focus from 0.690 to 0.766. They are that run’s median and 25th percentile. Cluster members below 0.690 are Expansion. Outliers are the pages HDBSCAN left as noise.

The centroid is the average vector of a cluster’s pages. I score similarity in the full 384-dimension space, not on the 2D map.

My first version set the thresholds per run, from quantiles of that run’s own distribution. That broke the metric. A quantile cut-off fixes each role’s share by construction, so about 75% of pages landed in Core plus Focus on every run, whatever happened to the content. With frozen lines, drift shows up as pages crossing them.

Core and Focus pages carry 58.8% of clicks. Treat swings of 2–3 points as noise: rounding the stored vectors alone once moved this number from 62.4% to 60.0%. And a role diagnoses fit, not value – Expansion and Outlier pages still earn about four clicks in ten.

Two clusters show what the roles say. Web development has 150 pages (with the second-language ones), and 78 of them sit in Expansion: a broad topic whose pages have spread toward neighboring ones. SaaS development has 20 pages, 17 of them Core: small, tight, and done. The first needs editorial structure. The second needs nothing.

Which pages to keep, grow, relocate, or prune?

Every page gets one of four actions from ordered rules, and the first rule that matches decides. Each threshold is my working rule, tuned on this site, so treat it as a starting point rather than a constant.

The four counts add up to all 2,436 URLs in the run. The 1,000-impression line is this site’s 75th percentile.

Relocate goes first, because every later rule assumes the page is judged inside the right topic. In Web development, 31 of 150 pages are relocate candidates – the progressive web apps post from the start of this article among them.

Prune is deliberately narrow, and it is never automatic. Thirty days without a click or an impression is a reason to look, not a verdict: the page may be new, seasonal, ranking too low, or short on data. Before any decision I check the page’s last 12 months, its age, its external links, and the role it plays for readers, and then choose between merging, redirecting, and leaving it alone. A merge needs its own evidence, which comes next.

I had to tighten the rules until they were selective. “No in-cluster link” applied to every page flagged 73% of the site, so it now applies to service pages only – 125 of the 228 grow pages. “Average position worse than 10” caught 2,164 of 2,456 pages with impressions, so grow now needs both real demand and striking distance.

Company news and history is the case the rules can’t make for you. Of its 50 pages, 47 come out as keep: the posts fit their topic, and the rules see nothing wrong. Whether to invest in a topic that carries no service page is an editorial call, and here mine is: measure it, don’t grow it.

How do clusters change content creation?

Measured clusters change content creation in three places: which internal links get suggested, which briefs carry an overlap warning, and which page pairs qualify for a merge.

Links – I add a same-cluster bonus of +0.05 to each candidate’s semantic similarity score, a value I picked by replay, not from theory. Replayed on 80 published articles, the bonus raised the in-cluster share of the top-8 suggestions from 51.5% to 71.0%, at a 0.9% drop in mean similarity. Service pages moved further, from 43.4% to 65.1%. That measures how well suggestions fit the clusters, not rankings or traffic. A +0.10 bonus reached 77.0% at a 2.1% loss – six more points for more than double the cost – so I kept 0.05. It’s a bonus, not a filter: a clearly better page from a neighboring cluster can still win. The same bonus applies to inbound link sources, and every source and target must be 200 and indexable in the latest crawl.

Where a bonus can’t help, it says so. For 49 of 145 grow service pages, the 30 nearest candidates included no page from their own cluster at all. Those pages need new content or a manual link, not a better ranking formula.

Briefs – a brief shows the planned page’s neighbors at similarity 0.6 or above. Below that it’s noise: a Kubernetes consulting draft was being shown branding posts at 0.16–0.28. Any existing page at 0.9 or above, roughly the top 5% of pairs, triggers an overlap warning. Two near-identical headless ecommerce posts sat at 0.905. Both cut-offs come from this site’s distribution, not from a standard.

Merges – similarity alone never justifies a merge. A pair also needs contested demand: for every query both pages appear for in Search Console, take the smaller of their two impression counts, add those up over 12 months, and divide by the total impressions of the smaller page. A score of 0.3 means 30% of the smaller page’s visibility comes from queries its sibling also shows for. Re-run for this post on the blog and glossary:

The trend runs one way: the closer two pages are in embedding space, the more often they split real queries. Search Console has never seen my vectors, so that’s a second independent check. The top two bands hold 25 and 17 pairs, so read them as a direction, not a rate.

My tiers, again my own rules: merge at similarity 0.85 or above with contested demand of 0.3 or above, watch at 0.1–0.3, review at similarity 0.9 or above with contested demand below 0.1. The surviving page is the one with the most clicks over 12 months, then impressions, then in-content links.

How often should you update topic clusters?

Only one job may recluster: a monthly full run, saved as a draft that I accept or reject. The weekly job never reclusters. It keeps every unchanged page where the accepted run put it – noise stays noise – and sends only new pages and pages whose text changed to the nearest centroid. A page joins that cluster only if its similarity clears the projection floor: the 10th percentile of member similarities in the accepted full run, 0.613. Below it, the page stays in noise until the next full run.

I need that split because a full run moves even when the site barely changes, as run 4 showed.

Change detection has to look at the text you embed. Screaming Frog’s HTML hash differed on 11,671 of 11,673 HTML pages between two crawls while the text was identical – the CMS changes the HTML on every response. Only 22 in-scope pages had really changed. The projection built on that hash marked all 2,486 pages as changed, and I rejected it. A second guard now refuses any weekly run in which more than 25% of known pages look changed: that signals broken detection, not a rewritten site.

Cluster identities survive re-runs. I match old and new clusters with the Hungarian algorithm on the Jaccard overlap of their pages: overlap of 0.5 or more inherits the old key, and a new cluster drawing at least half its pages from one old cluster is a split. Retired keys are kept, never deleted. From run 1 to run 2, 83% of pages kept their key, and across seeds 38 of 43 clusters matched at 0.5 or above (median 0.93).

Tried and rejected: weekly reclustering, the crawler’s HTML hash for change detection, recomputing role thresholds every run, clustering the second language on its own, and a forced split of commerce into seven subtopics.

FAQ

Survey first, then build

Keyword research still decides what to build next. It just shouldn’t go first on a site that already has thousands of pages. First you need to know which of them stand together, which links actually connect them, and which pages sit at the edge of their topic. Then the next keyword plan lands on a map that matches the site, and a new page doesn’t quietly split demand with an old one.

Survey the site you have. Then plan the one you want.

If your site has outgrown what a keyword spreadsheet can describe, this survey is where I start two kinds of work. In a content strategy, it becomes the topic map: your existing pages grouped into clusters, with the internal links and gaps between them. In a content audit, it feeds one decision for every URL, merge candidates backed by Search Console data, and a recrawl to confirm the changes landed. Tell me what you’re seeing.