Does continued pretraining on Russian legal texts improve ModernBERT encoders?

Two encoders, one fixed court-decision collection

The study asks whether continued pretraining on Russian legislative documents improves a Russian ModernBERT encoder on legal text. The starting point was deepvk/RuModernBERT-base; the adapted model, RuModernBERT-ruLaw, was produced by continued pretraining on the textIPS field of RusLawOD, a corpus reported to contain 304,382 legislative documents and 194,425,905 corpus tokens. The standard ModernBERT maximum input length of 8,192 tokens was retained.

The main comparison is deliberately narrow. Both the original and the adapted encoder were evaluated on the same fixed external collection of 1,031 court-decision segments, and both received the same hidden positions in each of five masking realizations. The collection differs from the training corpus in source and document type, but possible content overlap or paraphrased overlap was not specifically excluded, so the paper calls this evaluation external rather than independent.

One distinction matters before reading any number. The 194,425,905 figure counts corpus tokens, not the positions the model actually processes. The number of model-token positions depends on the tokenizer, the windowing, and the overlap, and the paper treats it as a different quantity. Documents were split into windows of up to 8,192 tokens with 512 tokens of overlap, so boundary positions appear in neighbouring windows.

Masked-token loss fell at 512, 2,048, and 8,192 tokens

The headline result is a lower mean masked-token cross-entropy for the adapted model in all 15 combinations of three maximum input lengths and five masking realizations. At maximum input lengths of 512, 2,048, and 8,192 tokens, the mean decrease was 0.10942, 0.07052, and 0.06604 natural-log units respectively. At 8,192 tokens the mean paired difference was 0.06604 nats.

The metric is worth stating precisely. Masked-token cross-entropy is the negative natural logarithm of the probability the model assigns to a hidden token's correct original value, averaged over masked positions. A higher probability for the correct token gives a lower loss. The paper's illustrative example uses hypothetical probabilities of 0.5 and 0.8, giving losses of about 0.693 and 0.223 nats; those numbers are teaching examples, not measurements from this run.

The reported 95% intervals summarize sensitivity to masking on this fixed collection. They do not quantify uncertainty across document collections, so they should not be read as a confidence statement about legal text in general.

Comparing lengths does not isolate distant context

It is tempting to read the three length results as a story about long context: the adapted model gains most at 512 tokens and least at 8,192, which might suggest that distant context matters less than expected. The paper explicitly blocks that reading. Comparing lengths does not isolate a causal effect of distant context because the prefix composition and the evaluated tokens also change with length.

The architecture explains why length is expensive in the first place. Full self-attention has quadratic cost in sequence length. ModernBERT alternates local attention, which connects a position to its neighbours, with global attention, which connects all pairs. At a fixed local-window width, doubling the sequence length approximately doubles the local pair count and quadruples the global pair count, ignoring boundary effects. The paper presents these as asymptotic scaling estimates, not hardware measurements.

The study provides no measured training duration, throughput, or peak memory, so the attention-cost table describes growth rates rather than a measured advantage of this particular run. FlashAttention reduces intermediate storage but does not remove the quadratic arithmetic in global layers.

Entity extraction could not separate the two encoders

The second evaluation addresses legal-entity extraction over five legal categories with numeric identifiers 2, 4, 9, 13, and 17. Character intervals were converted to BIO labels, and micro-averaged precision, recall, and F1 were computed using exact matches of token boundaries and entity class. Documents were processed in windows of up to 8,192 tokens with 1,024 tokens of overlap, and each window was scored as a separate sequence; predictions from overlapping windows were not explicitly merged at the original-document level.

The original and adapted models achieved entity-level F1 scores of 0.99852 and 0.99820. The adapted model is therefore slightly lower, not higher. The paper does not claim an NER advantage for continued pretraining, and it notes that a single fine-tuning run per model limits conclusions about the stability of this difference.

The more important problem is what the test could detect. Before splitting, rows were grouped by the SHA-256 hash of normalized text; of 97,668 rows, 15,783 duplicate rows and 109 rows belonging to 32 texts with conflicting annotations were removed, leaving 81,776 unique texts divided into 65,382 training, 8,133 validation, and 8,261 test examples. Exact texts do not overlap between splits. However, 99.95% of test spans have the same normalized surface form and class in the training split, so the evaluation provides limited evidence about transfer to previously unseen forms.

Masked-token loss and entity F1 answer different questions

The two evaluations are separate branches and their results cannot substitute for one another. The masked-language-model evaluation tests changes in hidden-token probabilities on a fixed collection. The span experiment evaluates a different task after fine-tuning, with its own windowing, its own overlap setting, and its own scoring rule.

The aggregation orders also differ. Training averaged the loss over all masked tokens in a minibatch. The external evaluation instead aggregated first over masked tokens within a fragment with a batch size of one, then with equal weights over fragments and masking realizations. Entity scoring used exact token-boundary and class matches, so moving an entity's beginning by one token makes the prediction incorrect.

This is why a lower masked-token loss and a marginally lower entity F1 can coexist without contradiction. They measure different things on different data with different denominators.

What the comparison does not establish

The comparison supports lower masked-token prediction loss for the studied pair of models and collection. It does not isolate the contribution of distant context, and it does not establish practical legal utility. The study evaluates model losses and one entity-extraction test; it does not assess legal expertise or practical labour savings.

The entity result is limited in a second way. Near-complete lexical coverage of the test split by training forms means the task mainly contains familiar forms, so it has limited discriminative value for distinguishing the two encoders. The paper also reports one fine-tuning run per model, which limits conclusions about transfer and about the stability of the small F1 difference.

The paper's own summary keeps three things apart: the reported language-modelling effect on a fixed court-fragment collection, the limited evidence from the entity test, and the untested questions of transfer to new forms, isolated distant-context contribution, and practical legal utility. No causal chain between these blocks is implied.

Read the loss gap as a bounded, consistent signal

For a reader deciding what this paper changes, the most defensible takeaway is the consistency of the masked-token result rather than its size. The adapted model's loss was lower in all 15 combinations of three maximum input lengths and five masking realizations on the fixed court-fragment collection, with a mean paired difference of 0.06604 nats at 8,192 tokens. That pattern is the paper's strongest evidence.

The entity-extraction branch should be read as a negative or at best uninformative result for this model pair. With F1 of 0.99820 for the adapted model against 0.99852 for the original, and with 99.95% of test spans already covered by the same normalized surface form and class in training, this experiment cannot show whether continued pretraining helps on unfamiliar legal entity forms.

The practical priority follows from that split. If the question is whether continued pretraining on Russian legislative documents improves masked-token prediction for this encoder pair on this collection, the answer is yes at every tested length. If the question is whether it improves legal entity extraction or delivers practical legal utility, this study does not answer it, and its own framing says so.

Sources used in this answer