How Should Web Crawlers Allocate Budget Across Hosts for LLM Pretraining Data?

Half of quality variation sits between hosts

The paper asks how a crawler with a fixed budget should divide fetches across websites, called hosts, and how deeply it should crawl each one, if the goal is to collect more pages that pretraining data filters would keep. The answer rests on one measurement: quality is partly a property of the host, not only of the individual page.

Using 3.01 million documents from 532,628 hosts extracted from 224 WARC files of the CC-MAIN-2024-10 crawl, the author scored each document with ten fastText classifiers and computed an intraclass correlation, ICC1. This statistic is the share of per-document score variance that lies between hosts rather than within them. Across the ten classifiers, ICC1 ranged from 0.43 to 0.62, so roughly half of the variation in scores is between hosts.

That matters because it means a host's pages are not a random mix. If a crawler learns something about a few pages from a host, it has learned something about the rest. The paper treats this as moderate consistency, not uniformity: a substantial share of variation still sits within hosts, which caps how well any host-level signal can rank hosts.

Two clarifications about the metric. Quality here means a classifier's predicted usefulness for pretraining, a proxy rather than intrinsic document quality or downstream benchmark performance. And the document counts describe the WARC sample, not all pages Common Crawl fetched from each host.

Centrality barely beats random host ordering

Common Crawl currently uses harmonic centrality, a web-graph metric, to guide which hosts to crawl and how deeply. The paper tests whether that choice increases the number of high-quality pages collected under a fixed budget.

In a crawl-yield simulation, hosts are fetched in the order a policy defines until a budget is spent. At a 10% crawl budget, ordering hosts by harmonic centrality collected 0.7 to 1.8 times as many high-quality pages as random ordering across classifiers, compared with 4.0 to 8.0 times for an oracle that knows each host's true quality density. PageRank and Common Crawl's current allocation also fell well short of the oracle. For DCLM, centrality-based ordering performed worse than random.

The paper offers a mechanism for the weak signal: language. A host's primary language covaries with its centrality, and English-trained classifiers score English and non-English hosts very differently, though not in one direction. DCLM rates non-English hosts far higher than English ones (median 0.29 versus 0.0005), so pooling languages makes its overall centrality correlation negative (ρ = −0.083) even though it is positive within English hosts (ρ = +0.064). Nemotron-Q leans the opposite way, scoring English hosts higher (0.49 versus 0.37).

The author is careful about what this explains. The language analysis does not establish that accounting for language improves crawl yield, and a positive correlation within a language group need not translate into an effective ranking under a limited budget. Centrality also roughly halved the number of top-level domains represented compared with random ordering, reducing diversity.

A three-page sample predicts a host's remaining pages

If quality is partly a host property, a small sample should forecast the rest. The paper tests this directly by correlating the quality density of a k-page sample with the quality density of the host's remaining pages.

The correlation rises with k and plateaus by about k = 15 to 20, but most of the signal arrives early: ρ = 0.53 at k = 3 and 0.57 at k = 5. A three-page sample therefore carries most of the information needed to rank hosts, at a fraction of the cost of scoring every page.

This is where the paper's policy proposal comes from. Sample-then-deepen probes three pages per host, then spends the remaining budget crawling deeply on hosts whose sampled quality density is highest. Because a crawler cannot tell a host's high-quality pages from its low-quality ones before fetching them, the policy can only choose which hosts to crawl deeply, not cherry-pick pages within a host.

The practical advantage is architectural: the classifier runs on a roughly three-page sample per host rather than on every candidate page, so it need not sit in the crawler's hot path. That distinguishes this host-level lever from page-level schedulers that must fetch, extract, and score every candidate to obtain a scheduling signal.

Sampling then deepening beats uniform depth by several times

The depth simulation gives each policy the same budget, a fixed number of pages to fetch, and scores it by yield: high-quality pages collected per page fetched. A page counts as high quality when its quality score exceeds 0.5.

Sample-then-deepen obtained up to 7.8 times more high-quality pages per fetched page than uniform breadth at small budgets, narrowing to 1.7 times as the budget grows, and approached the oracle policy. Depth biased by harmonic centrality, similar to what Common Crawl does today, barely differed from uniform breadth.

The gap shrinks as the budget grows because a larger budget forces the policy to descend into lower-ranked hosts. The comparison that matters for a crawler designer is therefore conditional on budget size: the advantage of sampling is largest when fetches are scarce.

The paper reports this result on its FineWeb-Edu scores and replicates the depth analysis at full scale using GneissWeb annotations covering every fetched page in CC-MAIN-2017-34. The full-crawl replication removes the WARC subsampling step but still reflects Common Crawl's own page selection, so the results do not establish that three pages suffice for arbitrary hosts or pages outside the observed crawls.

Common Crawl's crawl depth does not track quality

The paper then turns from simulation to the deployed crawl. Joining Common Crawl's host index, which reports known URLs, fetched URLs, and coverage per host, with GneissWeb quality annotations gives a per-host quality density: the fraction of a host's crawled pages that a strong quality filter kept.

Across 5,654,186 hosts in CC-MAIN-2024-10, the correlation between how fully a host is crawled and its quality density was −0.08, and the quality distribution did not rise with coverage. If anything, the least-crawled hosts were the highest quality. The pattern held across six crawls from 2021 to 2024, with correlations from −0.057 to −0.151, and coverage stayed uncorrelated with every classifier tested (|ρ| ≤ 0.07).

The author frames this as expected rather than a defect: Common Crawl was built as a broad, general-purpose sample of the web, not as an LLM-oriented crawler. The finding is that its per-host budget is not preferentially deepening the hosts worth deepening for pretraining data.

Missed high-quality pages concentrate on barely-crawled hosts

Assuming uncrawled pages resemble fetched pages from the same host, the paper estimates 58 million high-quality pages among known but uncrawled URLs in the CC-MAIN-2024-10 crawl alone. That is an estimate of scale, not an exact count, and it concerns known URLs rather than the whole URL population.

The missed budget is not spread evenly. Half of it lies on hosts crawled below 20%, and 62% falls on 369,534 hidden-gem hosts, defined as below-median coverage but top-decile quality, which represent only 6.5% of all hosts. The single densest coverage-by-quality bin holds 26% of missed high-quality pages. The skew runs toward national and regional news publishers and scholarly repositories; the .uk top-level domain alone accounts for 23% of the missed pages from 6.1% of hosts, at a median coverage of 34%.

Expressed as pretraining tokens, the single-crawl shortfall is on the order of tens of billions of high-quality tokens, roughly 58 billion at an assumed 1,000 tokens per page. Extrapolating to the more than 100 existing Common Crawl crawls would give approximately 6 trillion tokens, but the author labels this an illustrative extrapolation that does not account for repeated pages, duplicate content, or overlap across crawls.

Higher classifier yield is not the same as a better corpus

The paper's own limitations define where the recommendation stops. Quality density inherits the biases of whichever classifier produced it; the author mitigates this by showing the coverage result holds under independent models, but no model is ground truth. The missed-budget estimate also assumes uncrawled pages match crawled ones and ignores crawl politeness and content freshness.

More importantly, allocating budget to hosts with high classifier scores may reduce the representation of languages, regions, or content types that those classifiers score poorly. At crawl time this affects which pages are collected at all, including potentially useful pages on hosts whose initial samples score low. Higher classifier-based yield therefore does not necessarily imply a more representative or diverse corpus.

The paper also separates quality from discovery. The depth simulations allocate budget over a fixed set of already-fetched pages and do not measure which new URLs or hosts a crawler would discover by following links. A host with low scores could still link to useful content elsewhere, so weak performance as a quality-ranking signal does not rule out a role for centrality in discovery. Combining centrality-based exploration with sample-guided depth allocation is suggested but not evaluated.

Spend scarce fetches on sampled hosts, not central ones

The paper's decision rule is specific: when fetches are scarce, probe a few pages per host and put depth where the samples score highest, rather than ordering hosts by graph centrality or crawling all hosts equally. The advantage is largest at small budgets, up to 7.8 times more high-quality pages per fetch than uniform breadth, and it shrinks toward 1.7 times as the budget grows.

The second priority follows from the coverage analysis. The hosts most worth revisiting are not the well-crawled ones but the barely-crawled hosts with high observed quality density: 62% of the estimated 58 million missed high-quality pages sit on 6.5% of hosts. That concentration, not the aggregate shortfall, is the actionable finding.

What the paper does not settle is whether this yield gain survives the trade-off it names. A crawler that follows classifier scores may collect more pages its classifier likes while narrowing language and regional coverage, and the paper offers no evaluated mechanism for reserving exploration budget. The case for sample-then-deepen is therefore strongest where the objective is explicitly classifier-defined pretraining quality, and weakest where representativeness is a co-equal goal.

Sources used in this answer