💰 Information Gain

If we’re given more “information” about a distribution, 🔥 Entropy decreases. Specifically, for the distribution that has some correlation with , if we learn the value of , we narrow the range of results of and thereby decrease entropy.

Averaging over the probability of all that we can know, we get the equation for conditional entropy,

The decrease in entropy caused by us knowing is known as information gain.