How can differential entropy be approximated in Bayesian optimal experimental design?

Why the nested integral collapses to one

Bayesian optimal experimental design picks a design by maximizing expected information gain, the expected reduction in uncertainty about unknown parameters after seeing data. Written directly, that quantity is a nested integral: an inner average over the posterior or likelihood for each possible dataset, wrapped in an outer average over the prior and the data distribution. Each inner evaluation can require a likelihood call, which is exactly the expensive step in large-scale inverse problems where the forward map is a PDE solve.

The paper targets a narrower but common situation: the entropy of the likelihood is either independent of the design or explicitly computable, as happens for additive noise models whose noise entropy does not change with the design. Under that condition the design-dependent part of expected information gain is the differential entropy of the evidence distribution, so the nested integral collapses into a single integral. The design problem becomes a maximum-entropy estimation problem, and the question is how to estimate that entropy while touching the forward map as few times as possible.

A Gaussian mixture stands in for the evidence

The proposed estimator separates the two approximations that the nested formulation entangles. In the first loop, the method draws M points from the prior, either by plain Monte Carlo sampling or by randomized quasi-Monte Carlo points, and evaluates the forward map at each one. Those M forward-map outputs are then used to build a Gaussian mixture model surrogate for the evidence density: an equally weighted mixture of Gaussian components centered at the mapped points, with the noise covariance.

In the second loop, the differential entropy of that surrogate is evaluated with standard entropy methods, using N draws from the mixture. No further forward-map evaluations occur in this loop. This is the structural saving the paper claims: the expensive forward solves are confined to the M-point first loop, while the entropy evaluation reuses the surrogate. The authors note that the Monte Carlo scheme in the second loop is not the only option, and that the entropy estimator should be chosen to fit the application.

Convergence rates measured in forward solves

The paper's complexity comparison counts cost in forward-map evaluations, not in total runtime. With a Monte Carlo surrogate, the proposed method reaches a forward-solve complexity of order TOL^(-2-ω/η); with a randomized quasi-Monte Carlo lattice surrogate, order TOL^(-1-ω/η-ε); and with higher-order quasi-Monte Carlo of order α, order TOL^(-1/α-ω/η-ε). These last three rows come from balancing the K-dependent and M-dependent error contributions while neglecting the N-dependent cost of the final entropy evaluation.

The authors are explicit that these figures should be read as forward-solve complexity estimates rather than complete computational complexity estimates. The QMC rows also omit possible dependence of the QMC error constants on the approximation dimension K, so they are formal asymptotic forward-solve estimates unless those constants can be controlled uniformly as K grows. The comparison therefore holds in the regime where forward solves dominate, which is the regime the paper was motivated by.

How the rates compare with prior methods

Against the baselines listed in the paper, the Monte Carlo variant's order TOL^(-2-ω/η) matches the variational Monte Carlo approach of Foster et al., which the paper describes as O(TOL^(-2-ω/η)) under the same forward-discretization cost model, and improves on the standard nested Monte Carlo estimator at O(TOL^(-3-ω/η)). The randomized QMC lattice variant at O(TOL^(-1-ω/η-ε)) is comparable to the sparse-tensor QMC nested approach of Kaarnioja and Schillings and the multilevel randomized QMC approach of Bartuska et al., both listed at O(TOL^(-1-ω/η)).

The paper also distinguishes its assumptions rather than only its rates. It requires only mild smoothness of the forward map, whereas the dimension-robust QMC estimates of Kaarnioja and Schillings rely on substantially stronger weighted regularity and summability assumptions. The authors add that under those stronger assumptions their own evidence surrogate could inherit the dimension-robust inner-integration bound, suggesting the dimension dependence in their QMC constants comes from the weaker unweighted regularity framework rather than from the Gaussian-mixture construction itself.

What separates this from variational evidence approximations

The closest prior approach in the paper's account is the variational approximation of the evidence by Foster et al., whose evidence approximation converges at order O(M^(-1/2)) and whose Monte Carlo error converges at order O(N^(-1/2)), giving an overall root-mean-square error of O(M^(-1/2)+N^(-1/2)). The paper lists three distinctions. First, Foster et al. assume the target distribution can be represented in a finite-dimensional latent space, while the Gaussian mixture can approximate a non-parametric family of evidence distributions. Second, the proposed method is a direct approximation scheme that needs no additional computation beyond sampling and mapping the prior cubature. Third, quasi-Monte Carlo cubatures can exploit mapping properties of the model to accelerate the rates further.

Two experiments that isolate the M-rate

The numerical section is designed to isolate the convergence rate in M, the number of forward-map evaluations, by choosing N large enough that M-dependent terms dominate. The first experiment is a weighted deconvolution problem with a Gaussian prior and Gaussian noise in dimension d = K = 20, with prior standard deviation σ_x = 10, noise standard deviation σ_ε = 2, and kernel parameter γ = 0.1. Because the evidence entropy has the closed form J_K = (K/2)(1 + log 2π) + (1/2) log|AΣAᵀ + Γ|, the error of each estimator realization can be computed exactly. The reported root-mean-square error follows the predicted O(M^(-1/2)) for Monte Carlo and O(M^(-1+γ)) for randomized QMC, based on 30 realizations.

The authors also report a caveat about this first experiment: with those parameter choices the sufficient condition L₁² < (1/2)L₂ required by their theorem cannot be satisfied, so the theorem does not guarantee the rate there. They explain that for this linear Gaussian model the relevant expectation is finite anyway, which is why the predicted rate still appears. They further note, from undocumented experiments, that the Monte Carlo estimator's error can be closer to O(M^(-1)) for some linear Gaussian problems, possibly because the Gaussian mixture approximates Gaussian distributions very accurately.

The second experiment is a nonlinear elliptic PDE source problem on the unit square, where the unknown is a diffusion coefficient and pointwise solution values are the measurements, a setting the paper says can describe Darcy flow in a porous medium. Root-mean-square errors are compared against a reference entropy for three designs: the optimal design {(0.50,0.50),(0.75,0.50),(0.50,0.75)}, the worst design {(0.25,0.25),(0.25,0.50),(0.25,0.75)}, and the 42nd best design {(0.25,0.50),(0.75,0.25),(0.75,0.50)}, with N = 2^20 and a reference computed by a tent-transformed lattice rule with M₀ = 2^13. The same optimal design is rerun for truncation and mesh combinations (K, h) = (50, 1/64), (50, 1/128) and (100, 1/128), with rates consistent with the earlier figure.

Where the saving applies and where it stops

The method's advantage is conditional on structure that the paper states up front: the likelihood entropy must be design-independent or explicitly computable. When that holds, exploiting it and switching to maximum-entropy estimation is advantageous, and the resulting nominal rates are comparable to state-of-the-art approaches in the forward-solve-dominated regime while requiring milder smoothness assumptions. When it does not hold, the collapse to a single integral is not available and the argument does not carry over.

A second boundary is the cost model itself. The reported rates count forward solves and neglect the N-dependent cost of the final entropy evaluation, so they describe settings where that evaluation is negligible relative to the forward map. The authors also caution that their numerical examples use relatively simple forward operators, so they do not claim the first loop dominates computation times at the large N values tested; the experiments are presented as verifications of the M-rate rather than as demonstrations of wall-clock savings.

Within those limits, the paper's practical priority is clear: if a design problem has design-independent or explicitly computable likelihood entropy and expensive forward solves, the Gaussian-mixture evidence surrogate with quasi-Monte Carlo sampling is the variant to prefer, since it reaches the O(TOL^(-1-ω/η-ε)) forward-solve rate while keeping the entropy step free of additional likelihood calls. The Monte Carlo variant remains the fallback when the smoother QMC structure is unavailable, matching the variational baseline's rate rather than beating it.

Sources used in this answer