How can ensemble-based perceptual audio quality assessment provide confidence intervals?

Point estimates hide the uncertainty listening tests report

Standardised listening tests report a mean opinion score (MOS) of Basic Audio Quality together with confidence intervals (CIs) and statistical comparisons between conditions. Most objective audio quality metrics, by contrast, output only a single point estimate. The paper frames this gap as the motivating problem: a user of an objective metric cannot tell how uncertain a MOS prediction is, whether its interval would achieve adequate empirical coverage, or whether an estimated quality difference would lead to the same decision as a listening test.

The paper's stated primary objective is therefore not just a better MOS number but the conditional distribution of subjective listener scores for a given reference and signal under test (SUT) pair. Such a distribution would let a model simulate listening-test panels, estimate MOS uncertainty, and assess the likelihood of significant differences between conditions.

Eleven PEAQ features feed a bagging ensemble

The method is intrusive: for each reference and SUT pair, a non-trainable PEAQ-style front-end (ITU-R BS.1387) computes a feature vector of J = 11 model output variables (MOVs) describing perceptually relevant degradations such as bandwidth limitation, noise, and temporal or spectral distortions. These 11 features are computed once per signal pair and shared by all regressors.

That feature vector is passed to K low-capacity regressors. Each member is trained on a different bootstrap sample drawn with replacement from the training set D = {(z_i, m_i)}, where m_i is the observed subjective MOS for stimulus i, computed as the mean of its L_i listener scores on a 0 to 100 scale. The paper reports K = 50 base regressors as a practical accuracy-complexity trade-off, using Multivariate Adaptive Regression Splines (MARS) selected for their accuracy-complexity trade-off and smooth behaviour. Before calibration, the variation among member outputs reflects how sensitive the learned MOS mapping is to the bootstrap-resampled training data.

Calibration aligns ensemble spread with listener spread

Raw ensemble means can be biased or incorrectly scaled relative to subjective scores, and raw ensemble widths need not match subjective score variability. The paper therefore fits one global mean-and-width calibration on the training database. An affine transformation s'_k = a s_k + b maps the ensemble mean to mu' = a mu + b and scales the standard deviation by |a|; the parameters a and b are fitted by least squares against the observed MOS values.

A second step rescales the deviations of the affinely mapped member scores around their mean by a global width factor c_std, estimated as the median over calibration stimuli of the ratio of the subjective standard deviation to |a| times the ensemble standard deviation. This preserves the affine-calibrated arithmetic mean while scaling the member-score standard deviation. The calibrated member scores form an estimated empirical score distribution from which the point estimate (the mean of the calibrated members) and confidence intervals are computed.

Panel-matched CIs simulate listening panels of the same size

To compare the calibrated model distribution with a listening test containing L listeners, the method simulates B = 4000 pseudo-panels of the same size. For each bootstrap draw, calibrated member scores are sampled with replacement and averaged, and the nominal 95% panel-matched ensemble CI is given by the empirical quantiles of the resulting pseudo-panel means. Under the paper's artificial-listener hypothesis, this interval estimates the variation that would be obtained by repeatedly drawing panels of L calibrated model listeners.

The paper evaluates three increasingly demanding implications of that hypothesis: empirical MOS coverage, agreement of marginal score distributions, and agreement of paired significance decisions. Coverage is measured as the fraction of items for which the panel-matched ensemble CI contains the corresponding subjective MOS, with values closer to the nominal 0.95 indicating better coverage. Distributional agreement is measured by the mean per-stimulus two-sample Kolmogorov-Smirnov statistic D_KS, where smaller values indicate less worst-case mismatch between the subjective and ensemble cumulative distribution functions.

Calibration raises coverage without hurting MOS accuracy

Calibration improves coverage on every evaluated dataset: from 0.28-0.31 to 0.69-0.82 on the SEBASS-DB subsets, from 0.34 to 0.47 on ODAQ, and from 0.14-0.22 to 0.33-0.43 on USAC. It also reduces D_KS throughout, indicating better agreement between predicted and subjective score distributions. Coverage nevertheless remains below the nominal 0.95, particularly on USAC, which the authors partly attribute to the deliberate use of the publicly available SEBASS-DB for reproducible training at the expense of a mismatch with the codec, bitrate, and content conditions represented in USAC.

Affine calibration has little effect on point-estimate MOS accuracy; the small PCC and RMSE changes result from clipping member scores to the MUSHRA scale. On the common USAC1-3 sets, the calibrated ensemble reaches PCC/RMSE of 0.93/5.67, 0.80/8.39, and 0.77/9.52, whereas GML with CutMix reports PCCs of 0.88, 0.89, and 0.92 respectively. The paper notes that the method is competitive on USAC1 but weaker than GML on USAC2 and USAC3, while using only 11 fixed PEAQ features and low-capacity MARS regressors trained on the public SEBASS-DB (11 000 ratings), compared with GML's 67 505 individual ratings from an internal, codec-oriented corpus and a 15.25-million-parameter network trained on an NVIDIA A100 GPU.

Paired significance decisions become more conservative after calibration

For listening-test-style significant differences, listener scores are averaged across items for every listener-condition pair, a one-factor repeated-measures ANOVA with factor condition is computed, and paired t-tests with Bonferroni correction at alpha = 0.05 are applied to all condition pairs. The same procedure is applied to the ensemble, with members bootstrapped to the subjective panel size, and a pair is regarded as objectively significant when at least half of the bootstrap repetitions yield a Bonferroni-corrected significant difference.

Calibration makes these paired tests more conservative: the false-positive rate decreases or remains unchanged, but significant-pair recall decreases on every dataset. The paper explains this as a consequence of the width adjustment. For conditions A and B, calibration preserves their mean contrast while scaling the centered member variation in paired differences by c_std; because the fitted c_std is greater than 1, significance becomes less likely. The authors conclude that matching marginal score distributions is insufficient for paired decisions, which also depend on cross-condition covariance. These rates can also be unstable when based on few pairs: SAOC-DB has only 16 subjectively significant pairs, while USAC1-3 have only two, one, and one non-significant pairs respectively.

Where the uncertainty-aware metric transfers and where it does not

The paper's own scope statement is that the ensemble cannot yet replace panel-level hypothesis testing, but it does provide an uncertainty-aware objective audio quality metric: its mean estimates MOS, while its item-dependent spread and approximate CIs help identify predictions requiring subjective validation. The distributions also enable approximate pairwise comparisons, but not yet reliable significance inference.

The authors state that reliable listening-test inference would require separate calibration of marginal variability and cross-condition covariance using listener-level data, and that artificial listener panels should retain the current panel-size matching while also preserving within-listener dependencies and the target test design. Calibration and decision thresholds should then be validated on independent listening tests for CI coverage, false-positive control, and statistical power. Future work is described as investigating listener-level calibration of cross-condition covariance and condition-dependent uncertainty, and incorporating out-of-domain detection so the system can defer automatically when its uncertainty estimates are unlikely to transfer.

The practical priority is uncertainty screening, not significance testing

The most decision-relevant split in this paper is between two uses of the same calibrated distribution. For uncertainty screening, the calibrated ensemble is useful: coverage rises on every evaluated dataset, D_KS falls throughout, and MOS accuracy is preserved, so the item-dependent spread and approximate CIs can flag which predictions most need subjective validation. For paired significance inference, the same distribution is not yet reliable, because the global width factor c_std greater than 1 makes calibrated paired tests more conservative and reduces significant-pair recall on every dataset.

The paper's own priority ordering follows from that split: use the ensemble to target listening tests towards uncertain conditions, and treat the calibrated marginal distributions as a screening tool rather than a substitute for panel-level inference. The coverage gap that remains below 0.95, especially on USAC, is the clearest signal of where that screening value is weakest, and the authors tie it to the mismatch between the public SEBASS-DB training data and the codec, bitrate, and content conditions represented in USAC.

Sources used in this answer