Entropy
Also written: H(p)
The average surprise of a distribution, −Σ p log p. It measures how unpredictable the distribution is: a fair coin has 1 bit, a certain outcome has 0. It's also the lowest cross-entropy any model can reach.
Also written: H(p)
The average surprise of a distribution, −Σ p log p. It measures how unpredictable the distribution is: a fair coin has 1 bit, a certain outcome has 0. It's also the lowest cross-entropy any model can reach.