Distillation

Also written: distilled, student and teacher

Training a small 'student' model to imitate a large 'teacher' model's outputs, not just the labels. DistilBERT was made from BERT this way.