Glossary: Transformers Basics

Causal mask

Also written: causal attention, causal, mask

A True/False table that stops each token from attending to later tokens, by setting those scores to minus infinity before softmax. Without it, the model would copy the answer it is supposed to predict.