Glossary: Transformers Basics

Pretraining

Training on trillions of tokens of general text to predict the next token. The result is a base model.