Neural Weight Norm = Kolmogorov Complexity: The Hidden Logic of Weight Decay

TL;DR

Why is weight decay so effective? This paper provides a stunning answer: in any practical hardware environment (fixed-precision), minimizing the weight norm is mathematically equivalent to minimizing the Kolmogorov complexity of the model's output. By using weight decay, we are inadvertently performing "Solomonoff Induction"—the gold standard of optimal Bayesian inference.

The "Precision" Catch: Why Real-Valued Weights Failed Theory

For decades, researchers tried to link neural network "size" to "complexity." However, if you allow weights to be real numbers ($\mathbb{R}$), a single weight could theoretically store the entire library of Congress in its decimal expansion. In this "Super-Turing" regime, a network with a tiny weight norm could output a string of infinite complexity.

The author's core insight is that modern deep learning actually runs on fixed-precision (fp16, int8, ternary). In this discrete world, the norm of a weight vector is locked to the number of bits needed to describe it. This "Lp collapse" means that whether you use $L_1$ or $L_2$, you are essentially counting the number of non-zero parameters.

Methodology: The Two-Way Bridge

The paper proves a "Sandwich Bound" via two elegant reductions:

1. Programs to Networks (The Upper Bound): The author shows that any program for a Universal Turing Machine can be "injected" into a looped neural network. By using a specialized routing layer, each bit of the program costs exactly one non-zero parameter. Thus, the Neural Complexity $\mathcal{N}(s)$ is at most the Kolmogorov Complexity $K(s)$.
2. Networks to Programs (The Lower Bound): Conversely, any sparse fixed-precision network can be described as a list of (location, value) tuples. Since describing a location requires $\log (exttotalparameters)$ bits, the description length of the network is bounded by $W\log W$.

Deep Insight: Neural Prior vs. Solomonoff Prior

The most profound implication is the Solomonoff Corollary. Solomonoff's Universal Prior $M(s)$ is the "perfect" prior for prediction, but it is famously incomputable.

The author proves that the prior induced by weight decay ($Q(s)\propto \exp (-\lambda \Vert heta{\Vert }^2)$) matches the Universal Prior's exponent up to a logarithmic factor. This suggests that the reason weight decay works is that it guides the model toward "simpler" (more computable) hypotheses, similar to how an ideal Bayesian agent would operate.

Experimental Witness: The Permutation Example

Is the $\log$ factor just a mathematical artifact? No. The author demonstrates a "Permutation Matrix" example. A network can output a permutation of $N$ elements using only $O(N)$ parameters, but the complexity of that permutation is $O(N\log N)$. The network "cashes out" its address space to produce more information than it has parameters, proving the bound is tight.

Critical Analysis & Future Outlook

While the proof is conceptual and asymptotic (the constants $c_U$ and $c_d$ might be large in practice), it provides a rigorous theoretical grounding for:

• Quantization-Aware Training: Why lower precision often helps generalization.
• Looped Architectures: Why "Chain-of-Thought" or Universal Transformers are more efficient at representing complex algorithms.
• Sparsity: Why $L_1$ and $L_2$ decay eventually target the same structural information in quantized settings.

Conclusion: This work moves weight decay from a "fine-tuning trick" to a fundamental pillar of algorithmic information theory. It suggests that our current training recipes are much closer to "ideal induction" than we previously dared to believe.