Pretraining

Also written: pretrained, pretrained model

Training a model on huge amounts of general text before it sees your specific task. For BERT, this means guessing hidden words. The model can then be fine-tuned on a smaller labeled dataset, usually doing better than one trained from scratch.