Can multimodal flow matching and inference policy optimization reduce hallucination in speech enhancement?

Hallucination means changed words or a drifted speaker

Generative speech enhancement can sound cleaner and more natural than conventional discriminative systems, which predict masks, spectra, or waveforms under deterministic objectives. That expressiveness comes with a failure mode the paper treats as especially damaging for speech: a model can produce fluent, high-quality audio while changing words, inserting phonetic content, dropping weak speech, or drifting away from the speaker. A better-sounding output is therefore not automatically a better enhancement, because the errors corrupt exactly the content the system is supposed to preserve.

The paper's framing matters for how the results should be read. Hallucination here is not a single score but at least two separable components: lexical or semantic content, and speaker identity. The evaluation later maps these onto different metrics, so a system that improves perceptual quality while worsening word error rate has not solved the problem the paper poses.

Transcript conditioning alone does not prevent hallucination

An estimated transcript can supply linguistic evidence that is hard to recover from severely degraded acoustics alone, and prior work has injected linguistic or phonetic priors in various ways. But the paper states plainly that hallucination can occur even with transcript conditioning. That is the gap AuraSE targets.

The design response is a text-speech multimodal DiT: a double-stream-to-single-stream MMDiT in which a mel stream, carrying the flow state plus the degraded mel, and a text stream interact through joint attention before merging into single-stream blocks. In each double-stream block, mel and text tokens keep separate projections but attend jointly over the concatenated sequence; the attention output is split back into the two streams, each with its own output projection and feed-forward network. Only mel tokens reach the final projection head that predicts the velocity.

The paper's stated reason for retaining separate early representations is that the transcript is imperfect. Text can reinforce weak phonetic evidence without replacing the degraded acoustics that constrain speaker and signal characteristics. The paper reports that the double-stream MMDiT uses ASR and ground-truth transcripts more effectively than a single-stream design, which is the evidence behind the modality lever.

No single decoder configuration wins on every utterance

The second lever comes from an observation about inference. Diffusion and flow-matching models generate from a random latent and are sensitive to the classifier-free guidance scale, the sampling temperature applied to the initial latent, and the number of integration steps, yet enhancement research typically reports one fixed setting. The paper examines eight configurations and finds strong sample dependence: no configuration performs best for every utterance.

Per-utterance best-of-eight selection yields a substantially better quality and fidelity trade-off than each tested fixed configuration, but exhaustive per-utterance selection is too expensive to deploy. That tension is the problem IPO is built to resolve. The relevant quantity is not a single winning setting but the spread of outcomes across settings for the same input.

IPO turns inference diversity into training signal

Inference Policy Optimization is an online, on-policy preference optimization method. At each round, the current model decodes the same input under multiple inference policies; reward models score the outputs; sampled preference pairs populate a buffer; and the model is updated from their relative preferences. The updated model then generates the next round.

The multi-objective reward combines perceptual quality, WER, speaker similarity, and speech BERTScore. Pairs are enumerated where a positive reward gap exists, and one pair per utterance is sampled with probability proportional to that gap, which favors informative contrasts without always collapsing supervision to the single best-worst pair. Both members of a pair are model-generated; clean references enter only through ranking metrics, never as preference candidates.

The objective shares DPO's pairwise form but changes both data collection and the reference update. Offline DPO reuses candidates generated once and contrasts the policy against a frozen base. IPO instead contrasts the updated model with the previous-round model that produced the current buffer, and at the next refresh both candidates and reference advance with the model. The paper describes this as training on recent model outputs while limiting each round's update relative to its generating policy. Bounded buffer reuse amortizes rollout and reward cost over a number of updates, with frequent refreshes tracking the evolving model more closely and longer reuse eventually making pairs stale.

IPO improves content and quality together

The results fix the AuraSE base model and isolate RL post-training on a 300-utterance subset. Two reference points frame the comparison. The per-sample oracle, the reward-best of eight base policies per utterance, shows what static per-sample selection can offer (OVRL 3.271, WER 5.87%) and is explicitly non-deployable. GT-DPO, which pairs the ground-truth reference against a default decode, reduces OVRL and sharply degrades WER and SIM, reproducing the GT-winner collapse reported for generative speech-restoration alignment and underscoring the value of model-generated relative preferences.

Against those anchors the pattern is consistent. Holding the offline or online setting fixed, inference-policy candidates improve every metric: offline IPO beats DPO, and full online IPO beats online DPO. IPO also outperforms GRPO as an overall method. The key contrast is directional: DPO and GRPO raise perceptual quality but worsen WER, trading content faithfulness for noise suppression, whereas IPO improves OVRL and WER together and is best on every metric among the trained methods. IPO also surpasses the best-of-eight base reference on OVRL, SIM, and SBS, showing gains beyond static per-sample selection.

A post-hoc diagnostic on a randomly sampled 20-utterance subset with 8 candidates per utterance and a reward maximum of 10 reports a reward spread of 0.58 and mean reward of 7.38 for IPO, versus 0.53 and 7.32 for DPO and 0.54 and 7.25 for GRPO. The paper reads this as inference-policy variation yielding a larger reward spread and higher mean reward than stochastic reseeding or SDE perturbation.

Training dynamics favor a properly sized buffer

The optimization trajectories add a practical detail. In the comparison, an intermediate buffer has the smoothest EMA reward trajectory and reaches the highest reward, while an over-large buffer retains stale pairs and settles lower. Buffer-free IPO still beats GRPO and online DPO, which the paper takes as support that inference-policy candidates are the main source of the gain.

The reward spread among rollout policies shrinks during training, consistent with distilling configuration-dependent gains into one model, although the paper notes that reduced output diversity may also contribute. That alternative explanation is left open rather than resolved.

Deployment uses one fixed decoder

The headline deployment claim is deliberately narrow. At deployment, AuraSE-IPO uses a fixed 10-step ODE decoder without classifier-free guidance and without per-utterance configuration search. The multi-configuration exploration is a training-time device; the deployed system captures the gains it exposed through a single setting.

This distinguishes IPO from the closest flow-matching baselines. Offline DPO keeps the deterministic ODE sampler but its preference corpus is generated once, so static candidates may become less representative as the model evolves. GRPO-style training collects SDE rollouts and optimizes transition likelihoods, and FlowSE-GRPO explores through stochastic perturbations of one decoder configuration and recomputes transition likelihoods during training, training with an SDE but returning to the original ODE for deployment. IPO uses ODE outputs throughout and requires no trajectory likelihood.

Where the evidence transfers and where it stops

The paper argues its evidence is not limited to reward scores. Improvements transfer to held-out synthetic conditions, previously unseen real recordings, and an independent input-referenced blind test that does not use the training reward models and explicitly evaluates semantic and speaker hallucination. AuraSE-IPO ranks first on 11 of 12 metrics across the synthetic test sets and obtains the highest DNSMOS and blind-listening scores among the evaluated systems on the real DNS blind test set.

Several conditions bound those claims. For ASR, the transcript is generated once from each noisy input and fixed across AuraSE variants, while the same Whisper decoding protocol evaluates every system output, so dependence on an external recognizer remains. WER is scored against the reference transcript rather than the conditioning transcript, so copying an erroneous ASR condition is penalized rather than rewarded. Online RL remains costly because each rollout requires vocoding and metric evaluation, and the authors scope their claims to flow-matching speech enhancement.

The most decision-relevant distinction in this paper is not generative versus discriminative, but which training signal a generative enhancer is optimized against. DPO and GRPO raise perceptual quality while worsening WER; IPO improves OVRL and WER together and beats the non-deployable best-of-eight base reference on OVRL, SIM, and SBS. For a reader weighing hallucination risk, the priority is that content and identity metrics move in the same direction as quality, and that this holds under a single fixed 10-step ODE decoder rather than per-utterance search.

Sources used in this answer