🪙 Bayes' Theorem

Given events and , the following always holds.

Info

This is commonly used to reverse the conditionals or incorporate extra probabilities that we can find.

In bayesian terms, each element in the equation has a name:

The posterior is our belief for given evidence . The likelihood is the reverse, the probability of given a fixed . Finally, the prior is our initial belief for .

The evidence can be computed as follows:

Thus, it can be interpreted as the expected likelihood, taking the expectation over the prior. Note that this integration is difficult to compute algorithmically.

Conjugacy §

Note that the posterior is proportional to the prior multiplied by likelihood. This product can be difficult to compute analytically, so we often use conjugate priors, defined as priors with the same form as the posterior.

These priors come from the 👨‍👩‍👧‍👦 Exponential Family of distributions, with the most common one being the 👑 Gaussian.