Glossary: Transformers Basics

Grouped-query attention (GQA)

Also written: GQA, grouped-query attention

A variant where groups of query heads share one set of keys and values. It uses much less memory when generating text.