Can Bayesian speech synthesizers learn from multiple teachers?

Why one text needs a spread, not a point

Text-to-speech is a one-to-many mapping: one sentence can be spoken with different timbre, prosody and rhythm, and the same speaker reads it differently on each repetition. The valid realizations therefore form a distribution, and how wide that distribution is depends on the text and its context. A model that returns a single output, or a fixed amount of randomness, cannot represent that input-dependent spread.

The paper groups existing systems into three families. Discrete codec-based autoregressive models predict tokens, which introduces quantization error. Diffusion models generate continuous features by iterative denoising. Continuous-valued autoregressive models predict mel-spectrogram frames one at a time, either with diffusion or flow-matching heads or, as in MELLE, with a Gaussian whose KL loss is minimized when the variance equals one for every frame regardless of input. In all of them the spread is either implicit in sampling or fixed to a constant.

BELLE predicts a variance prior for every frame

BELLE (Bayesian evidential learning with language modelling) keeps MELLE's architecture and single-pass sampling and changes only the layer that outputs each frame. Instead of fixing the variance, it places a Bayesian prior over the mean and variance of each mel-spectrogram frame and predicts the parameters of that prior from the input. The spread is thus estimated for each input rather than set in advance.

The paper is explicit that this is evidential deep learning, not a Bayesian neural network: the network weights are point estimates, and the prior sits over the parameters of each frame's likelihood. Training maximizes the marginal likelihood of the observation with the latent parameters integrated out in closed form. The practical consequence is that the model outputs a distribution over frames, and the width of that distribution is a prediction, not a hyperparameter.

Estimating variance requires repeated recordings

Accurate variance estimation theoretically needs multiple observations per input, while standard TTS corpora provide one recording per text. The authors therefore build two complementary sources of one-to-many supervision. ParallelVox is a human parallel-speech corpus assembled from five public datasets, in which every text prompt is paired with multiple recordings. LS+6TTS augments LibriSpeech with speech from six pre-trained TTS teachers, giving each text multiple realizations.

Both BELLE and MELLE are trained on LS, LS+6TTS and the ParallelVox training set, using every realization of a text as a target. To compare real and synthetic parallel speech at equal volume, BELLE is also trained on LS+6TTS-881h, a text-sampled subset of LS+6TTS that matches the ParallelVox training set in hours while keeping all seven realizations of every retained text.

BELLE beats MELLE under matched training conditions

On LibriSpeech test-clean, intelligibility is measured by WER-H, the word error rate of a HuBERT-based ASR model on generated speech, and speaker similarity by SIM-o, the cosine similarity between WavLM-TDCNN speaker embeddings of generated and prompt speech. Two inference conditions are used: Continuation, where the first 3 s and the transcription prompt the rest of the utterance, and Cross-Sentence, where a prompt sentence is used to say a different sentence in that voice.

Against MELLE trained with the same backbone, data and schedule, BELLE's advantages are consistent. On LS, all five objective differences favor BELLE with p ≤ 0.004; on LS+6TTS, all seven differences favor BELLE with p < 0.05. The largest Cross-Sentence WER-H gap is on LS, where BELLE is 3.75 points lower. On ParallelVox, the two SIM-o differences and the diversity difference favor BELLE with p ≤ 0.007, while the two WER-H differences are not significant. MOS and SMOS, collected only on LS+6TTS, also favor BELLE, by 0.19 (p = 0.021) and 0.33 (p < 0.001).

A free variance alone does not explain the gain

A natural objection is that any learned variance would help. The ablation on LS-100+2TTS separates the sampling module from its loss. With no sampling module, Continuation and Cross-Sentence WER-H are 5.00 and 13.07. MELLE's Gaussian with a KL term toward unit variance reaches 3.24 and 6.83. A Gaussian with a free variance trained by negative log-likelihood reaches 3.88 and 7.31, trailing MELLE in both conditions and trailing BELLE by 2.5 points of Cross-Sentence WER-H. BELLE's NIG evidential loss reaches 2.66 and 4.84.

The authors offer two properties of the free-variance loss as explanation. Its gradient with respect to the mean is scaled by 1/σ², so frames assigned a large variance contribute little to learning the mean. It also fits each frame's variance to the few realizations available: LS-100+2TTS has three per text, and on real speech with three readings most frame-to-frame variation of such a variance is sampling noise. BELLE's Inverse-Gamma prior instead pulls that estimate toward a prior mean.

Predicted variance tracks variation across human readings

The paper tests whether real speech actually has the properties the prior assumes, using aligned human readings alone. Across 114 texts with at least three readings, the ratio of a frame's 99th to 1st percentile standard deviation has a median of 3.2 (interquartile range 2.7 to 3.6); if variance were constant across frames, sampling noise alone would give 1.30. A shrinkage estimator puts about 79% of the weight on the global variance with three readings, implying that roughly 79% of the frame-to-frame variation in empirical variance is sampling noise, a share that falls as readings grow.

The model's predictions follow the same pattern. Teacher-forced on the medoid of each group in the single-speaker repeat test set, BELLE's predicted variance correlates with the variance across readings with a median Spearman correlation of 0.18 (95% CI 0.17 to 0.18), positive in 88% of groups and significant in 57%; the median partial correlation after removing spectral flux, energy and alignment irregularity is 0.17. In a mismatched control pairing one sentence's predicted variance with another sentence's readings, the median correlation is 0.01 and no group is significant. The median slope is 0.069 for BELLE versus 0.024 for MELLE, a paired difference of 0.044 that is positive in 89% of groups.

Synthetic teachers match real parallel speech at equal volume

The data-side comparison is the paper's most consequential practical result. Parallel speech improves BELLE over single-recording data of similar volume, and at the same volume synthetic parallel speech performs comparably to real parallel speech while being far easier to scale. The LS+6TTS-881h subset exists precisely to make that comparison at matched hours, with all seven realizations retained per text.

The teacher weights were also tuned. On LS-100+6TTS, weights of 0.22 for the human recording and 0.13 for each teacher realization gave the lowest WER-H in both conditions, while 0.16/0.14 gave the highest SIM-o; raising the human weight above 0.22 worsened all four metrics. Because the SIM-o differences between those two settings are at most 0.006 while the WER-H differences are 0.07 and 0.44, the 0.22/0.13 setting is the one used.

Where the evidence stops

BELLE is trained and evaluated only on English speech, so behavior on other languages and cross-lingual synthesis is untested. The official MELLE implementation is not public, so every MELLE number comes from the authors' re-implementation with the same backbone, data and schedule; absolute scores may differ from those originally reported. The realization-weight comparison and the sampling-module and flux-loss ablations run on corpora built from a 72 h LibriSpeech subset, and whether the same rankings hold at LS+6TTS scale is not tested.

Scale is the sharpest boundary. The largest training corpus here is LS+6TTS at 4.8k h, while the larger systems BELLE is compared against on LibriSpeech test-clean objective metrics were trained on 44k to 250k h. How BELLE behaves beyond this volume, for instance when parallel speech is synthesized for a much larger corpus with one recording per text, remains untested.

Within those bounds, the paper's priority is clear: the value comes less from adding a variance head than from pairing an evidential prior with parallel supervision. The free-variance ablation shows a learned spread alone trails MELLE, and the equal-volume comparison shows synthetic teachers can supply that supervision at a scale real repeated recordings cannot easily match. Streaming generation, which BELLE supports because each frame costs one forward pass and is sampled in closed form, extends the same design rather than changing the comparison.

Sources used in this answer