Glossary: Transformers Basics

BPE (byte pair encoding)

Also written: BPE, byte pair encoding

A tokenizer that starts from single characters (or bytes) and repeatedly merges the most frequent neighboring pair into a new token. Common words end up as one token and rare words are built from pieces.