KL divergence

Also written: Kullback–Leibler divergence, KL

Cross-entropy minus entropy: a measure of how far a predicted distribution is from the true one. Training can shrink it but never the entropy itself.