KL divergence measures the difference in two probability distributions and . If is the "true" distribution and is the "approximation," it measures

The third form is exactly the difference between 💧 Cross Entropy and 🔥 Entropy. We can interpret this value as the expected extra number of bits to transmit using our predicted instead of true . It's equal to if .

Info

Note that this value is not a true distance: it's non-symmetric and does not satisfy triangle inequality. However, it is non-negative.

Reverse KL Divergence#

Reverse KL divergence, as its name suggests, measures

The key difference is the following:

1. In forward KL, is penalized for being too small when is large—failing to cover . Here, KL is mass-covering.
2. In reverse KL, is penalized for being too large when is small—failing to match dominant modes of . Here, KL is mode-seeking.

Applications#

We commonly see KL divergence or cross entropy used as loss functions in categorization problems (for example, in 🦠 Logistic Regression). The truth label is a one-hot encoding, and our prediction consists of softmax probabilities. In this case, our KL divergence simplifies to

for a single datapoint where is the true label. Moreover, since the entropy of the true labels is constant regardless of our model's predictions, we can instead just use cross entropy as our loss function.

We can also apply KL divergence to 💰 Information Gain. Rather than computing it as the difference in entropies before and after knowing , it can instead be interpreted as the difference in distributions