Masked language modeling
Also written: masked LM, MLM
Training by hiding some tokens and having the model fill them in, as BERT does. Labels of −100 mark unhidden positions so the loss ignores them.
Also written: masked LM, MLM
Training by hiding some tokens and having the model fill them in, as BERT does. Labels of −100 mark unhidden positions so the loss ignores them.