Skip to content

Glossary: Transformers Basics

Smoothing

Adding a small count, such as 1, to every possible pair so that none has probability zero. This prevents an unseen pair from making the loss infinite.

Where the book teaches it

Related terms

Bigram modelCross-entropy loss