Glossary: Running Open Models Locally
Attention layers
The layers of a transformer model that look back over earlier tokens. Each one stores keys and values in the KV cache, so the number of attention layers drives the cache size.
Glossary: Running Open Models Locally
The layers of a transformer model that look back over earlier tokens. Each one stores keys and values in the KV cache, so the number of attention layers drives the cache size.