Cross-entropy loss

Also written: cross-entropy

The standard loss for next-token training: the average, over tokens, of minus the log of the probability the model gave the correct token. Taking e to the power minus the loss gives the geometric mean probability of the correct tokens.