Multi-head attention
Several smaller attention heads running side by side, each with its own queries, keys and values, whose outputs are joined and mixed. It costs about the same as one wide head.
Several smaller attention heads running side by side, each with its own queries, keys and values, whose outputs are joined and mixed. It costs about the same as one wide head.