Weight tying
Using one matrix as both the token embedding table and the language-model head, so the vector for 'e' going in is also used to score 'e' coming out. It saves parameters.
Using one matrix as both the token embedding table and the language-model head, so the vector for 'e' going in is also used to score 'e' coming out. It saves parameters.