The evidence lower bound (ELBO) is a lower bound on the log probability of our data . This is often used to find an accurate variational distribution or maximize the likelihood , in context of 🧬 Variational Inference.

Consider a latent variable model involving observed data and latents . The evidence (or variational) lower bound (ELBO) deals with a joint distribution on observed data and latents . Specifically, it lower bounds the evidence, defined here as the log likelihood of ,

where is a distribution with parameters that we use to approximate .

Relation to Entropy#

By rearranging terms, we can easily see that

We can interpret this intuitively as the first term finding that maximizes (ie, finding the peak), and the second 🔥 Entropy term widening the distribution (ie, expanding the peak).

Relation to KL Divergence#

Another derivation starting from ✂️ KL Divergence between and gives us

where ELBO is the above bound. This means that the gap in the inequality is exactly the KL divergence between approximate posterior and the true posterior .

From the above expression, we can see that maximizing the ELBO can achieve two objectives: maximizing likelihood and minimizing KL.

1. Maximizing likelihood is useful when we want to find the optimal given the observed variables we want to model.
2. Minimizing KL, equivalent to finding the best approximation , is commonly used in variational inference.