RoPE (rotary position embedding)
Also written: RoPE, rotary position embedding
Rotating each query and key by an angle proportional to its position inside attention, so scores depend only on how far apart tokens are. Llama-style models use it.
Also written: RoPE, rotary position embedding
Rotating each query and key by an angle proportional to its position inside attention, so scores depend only on how far apart tokens are. Llama-style models use it.