Same Error, Different Function: The Optimizer as an Implicit Prior in Finance

TL;DR

In the world of S&P 500 volatility forecasting, a "Leaderboard Tie" is the norm. Whether you use a Transformer or a simple Linear Regression, the test error (NMSE) often looks identical. However, this paper reveals a startling truth: models with the same error are NOT the same models. By looking under the hood, the authors find that the optimizer (SGD vs. Adam) dictates whether a model becomes a stable, low-turnover predictor or a reactive, high-turnover one.

Academic Positioning: This work moves the conversation from "which architecture is better?" to "how does the optimization geometry select the function?"—positioning the optimizer as a first-class modeling component rather than a background utility.

The Problem: The "Rashomon Effect" in Finance

Financial data is notoriously noisy. In this "low signal-to-noise" regime, we encounter the Rashomon Effect: a phenomenon where many different models (the "Rashomon Set") fit the data equally well.

Current benchmarks focus almost exclusively on thin margins of loss improvement. But if two models have the same loss, are they interchangeable? The authors argue No. Existing evaluation pipelines are underspecified; they don't tell us how the model reaches its conclusion, only that it got the "answer" (the loss) right on average.

Methodology: Visualizing the Invisible

To prove that models are diverging functionally despite having identical errors, the authors employed three "Functional Diagnostics":

1. Impulse Response Analysis: Stress-testing the model by spiking a single input lag and watching the output reaction.
2. Difference Surfaces: Mapping the literal subtraction of one model's output from another across the input space.
3. SHAP Attribution: Checking which "days in the past" the optimizer decides are important.

The Optimizer as the Architect

The core discovery is that the optimizer chooses the curvature of the solution.

• SGD is "attracted" to flatter regions of the loss landscape, leading to simpler, nearly linear functions.
• Adam and Muon (adaptive methods) can navigate into sharper "valleys," allowing the model to learn complex, sigmoidal dampening effects for extreme shocks.

Figure 1: While NMSE is tied, SGD (Green) learns a flat response, while Adam (Blue) learns a complex nonlinear curve.

The Mechanism: The Edge of Stability (EoS)

Why does this happen? The authors point to the Edge of Stochastic Stability (EoSS). Neural network training naturally "sharpens" until it hits a stability limit defined by the learning rate ($\eta$).

• SGD hits this limit and stabilizes in flatter regions.
• Adaptive methods use preconditioning to effectively "flatten" the landscape, allowing them to descend into much sharper minima (2-5x sharper than SGD) that encode higher-order nonlinearities.

The authors proved this through a "Swap" experiment: taking a converged Adam model (complex) and switching the optimizer to SGD. The model immediately "collapsed" into a simpler SGD-like function, proving the optimizer's geometry is a constant force-field shaping the function.

Figure 2: Curvature (${\lambda }_{max}$) rises until it hits the theoretical stability threshold, where the optimizer's constraints begin to dictate the function's "simplicity."

Financial Implications: The Sharpe-Turnover Frontier

This isn't just an academic exercise. When these "identical" models are used to build portfolios:

• Performance Equity: Most models achieved similar Sharpe Ratios.
• Implementation Divergence: Adam-trained models produced significantly higher turnover (trading frequency).

Essentially, an adaptive optimizer "buys" into the noise more aggressively, leading to a "reactive" strategy. SGD "buys" into a more stable, long-term trend. If you have high transaction costs, the SGD model is a winner; if you have zero costs, the Adam model might capture more nuance. The optimizer, not the researcher, made that economic choice.

Figure 3: Moving vertically on the y-axis (Turnover) without moving on the x-axis (Sharpe) demonstrates that the optimizer determines implementability even when accuracy is tied.

Critical Analysis & Conclusion

This paper is a wakeup call for Financial ML. It suggests that:

1. Leaderboards are deceptive: A tie in NMSE/R² hides massive differences in how a model will behave in a real trade.
2. Ensembling works: Because different optimizers find "orthogonal" errors, averaging an Adam model with an SGD model actually reduces error more than tuning either one further.
3. The Optimizer is the Prior: We must stop choosing optimizers based on training speed alone. We should choose them based on the Inductive Bias we want (e.g., "I want a stable, low-turnover model, therefore I will use SGD").

Final Takeaway: In the Rashomon regime of finance, the question isn't "which model is right?" but "which function matches my economic objective?" If you don't choose, the optimizer will choose for you.