Can policy regularizations make deep RL inventory management easier to tune?

Base Stock structure narrows what the network must learn

An off-the-shelf deep reinforcement learning (DRL) inventory policy is typically a single neural network that reads the current inventory position and contextual features and outputs an order quantity directly. The paper's regularizations instead fix the shape of that mapping. Under the Base regularization, the network outputs a target level of total inventory, and the order quantity is derived from that target through the tot(It) term. Under the Coeff regularization, the order quantity is written as learned coefficients multiplied by five demand-related features: short- and long-term historical demand, short- and long-term forecast demand, and a constant bias. The Both variant combines the two.

This matters because the action stored in each training transition changes meaning. With Base regularization the action represents a target inventory level rather than a final order quantity, which constrains the inventory state to a narrower range consistent with a Base Stock policy. The paper reports that under a hyperparameter configuration tuned for unregularized DDPG in Setting 1, the critic loss under Base regularization is consistently lower over the first 10,000 timesteps, and the visited states sit in a narrower band. The authors' explanation is that each state in that band is visited more often, so sampling and Q-function learning are more effective. This is the paper's proposed mechanism, not a separately isolated causal test.

Regularization cuts the trials needed to beat DeepStock

The tuning claim is stated in countable terms. The paper reports that with regularization, the number of hyperparameter trials needed for the traditional DRL methods DDPG and PPO to outperform DS falls to 11 and 7 respectively. The same passage frames this as a large reduction relative to what those methods require without regularization, and the authors connect it to their Takeaway II: regularization improves the final testing loss of DDPG and PPO by a greater amount than it improves DS, which can flip whether DDPG or DS is best in Setting 3.

The mechanism offered for the larger benefit to traditional methods is specific to how they learn. DDPG minimizes Bellman error over stored transitions (s, a, r, s'), and the action a in each transition is the output of the policy network. Once Base regularization redefines that output as a target inventory level, the transitions cover a tighter state range, which the paper links to faster Q-function convergence. DS, by contrast, takes policy gradients directly on trajectory losses in the training set, so it does not learn from those transitions in the same way.

On Tmall offline data, DDPG with both regularizations wins

The offline evaluation uses 50,000 randomly selected SKU trajectories from July to September 2024 for training, 5,000 other SKU trajectories from the same 90-day horizon for validation, and 5,000 SKU trajectories from February to April 2025 for testing. Context vectors lie in R^190, review periods are typically 4 or 7 days, and lead times are typically 5 or 7 days. Policies were trained on an objective in which a 1% decrease in Stockout Rate is worth roughly a 2-day decrease in Turnover Time, and results are reported as differences from the DDPG-with-Both policy, where positive values mean worse.

In that table, DDPG with Both regularizations is the best DDPG version, and DS with Base is the best DS version. Relative to DDPG-Both, unregularized DDPG is 10.10 percentage points worse on Stockout Rate and 6.13 days worse on Turnover Time, while DS-Base is 2.18 points worse on Stockout Rate but 2.81 days better on Turnover Time. Because Stockout Rate is weighted twice as heavily as Turnover Time, the paper concludes DDPG-Both is comfortably better than DS-Base and DS-Both. The authors also note these results come after 10 sequential hyperparameter trials over 10 physical days, with one DDPG trial taking about 24 wall-clock hours at this data scale, so the comparison reflects a realistic rather than exhaustive search.

DeepStock trades sample efficiency for in-sample fit

The paper does not present DS as uniformly weaker. In Setting 3 with Base regularization, DS has better validation loss than DDPG but worse testing loss, which the authors attribute to overfitting: DS takes policy gradients directly with respect to trajectory losses in the training set, forcing in-sample minimization, and because it does not learn from (s, a, r, s') transitions it lacks the cross-temporal learning central to traditional DRL. The overfitting appears to subside when the sample size rises from 10 to 20, and DS performs best in Setting 2.

A separate experiment trains DS with Base regularization across sample sizes and horizon lengths. With only 5 training samples, the testing-loss gap remains about 8% even at horizon T = 129 under a simple IID distribution, so lengthening the horizon yields limited improvement. Raising the training set from 5 to 10 to 20 samples improves testing losses across all horizon lengths, which the authors read as evidence that DS does benefit from meta-learning across SKU trajectories, though convergence is not observed at longer horizons. With limited samples, DS also shows a large validation-to-testing gap.

Deployment scale rests on a deliberately simplified model

The deployment evidence is large in scope. As of October 2025, the algorithm manages inventory replenishment for 100% of products, domestic and international, sold by Alibaba on its B2C platform Tmall, covering over 1 million SKU-warehouse combinations. Alibaba owns inventory and sales for over 100,000 SKUs, controlled separately at roughly 20 primary warehouses, and aims to retrain every 1.5 months using the most recent 90 days of data. The paper contrasts this with earlier RL implementations that required grouping similar products and training separate models per group, each with its own feature pipelines, actor/critic networks, and tuning process; the regularized version is described as a single general-purpose model across all products.

That scale comes with stated model simplifications. Training relaxes order quantities to a continuous space although real decisions are typically integer-valued, and the policy is trained on a simplified inventory dynamic with deterministic lead time L. The authors acknowledge that if actual lead time is highly random, a policy calibrated for a deterministic L may fail to maintain sufficient inventory, and that decisions may perform poorly under model violations. They also note that real deployment has non-deterministic review periods and lead times, and that demand during stockouts must be inferred by extrapolating from the time of stockout. Their position is that training better on a simpler model outweighed modeling more complex dynamics, and the deployment outcome is offered as support for that choice rather than as a controlled test of it.

Treat the policy form as part of the tuning budget

The paper's most decision-relevant result is a comparison, not a universal ranking: the best DRL method for inventory management depends on whether policy regularizations are in place. Without regularization, DDPG was much worse than DS on the Tmall offline test; with Both regularizations, DDPG-Both beat DS-Base on the stockout-weighted objective, and the trial counts needed to reach that point fell to 11 for DDPG and 7 for PPO. A team comparing DRL methods on inventory data should therefore expect the ranking observed with plain off-the-shelf implementations to shift once Base Stock structure is imposed, and should budget tuning effort accordingly.

The boundary is equally specific. The offline comparison is a 10-trial sequential search on 90-day Tmall data with 190-dimensional context, and the authors note that parallelizing the search could reduce the improvement of DDPG-Both over unregularized DDPG while only increasing DDPG's advantage over DS. DS remains the better choice when samples are plentiful, as in Setting 2, and its weakness appears when training trajectories are scarce. The deployment claim covers over 1 million SKU-warehouse pairs under a simplified model with deterministic lead time, so the strongest support is for large-scale, single-model replenishment where the policy form can absorb product diversity, not for settings where lead-time randomness dominates.

Sources used in this answer