Glossary: Transformers Basics

Parameters

Also written: parameter count, weights

The learned numbers inside a model, such as embedding tables and linear-layer weight matrices. They can be counted with `sum(p.numel() for p in model.parameters())`; a tiny GPT has about 163,000 and GPT-2 small about 124 million.