Variational inference is a technique for approximating complex distributions with a latent variable model,
where is a latent not present in the dataset. We design and to be simple distributions (usually 👑 Gaussian), and composing them together allows us to represent much more complex distributions.
To optimize such a model, we find
Intuitively, this means we want to "guess" the given and maximize their probabilities. However, it's unclear how to calculate because this distribution is usually very complex. Variational inference addresses this by approximating
In other words, we'll estimate this complex distribution with a simple one, and we can show that even with this approximation, we can bound as
Usually, there are way too many datapoints to have an exact for all . Instead, we amortize by learning a model , sometimes called the encoder, which brings us to the 🖋️ Variational Autoencoder. This fundamental idea underlies many concepts, including 🕯️ Diffusion models and 🎛️ Control As Inference.