Glossary: Serving LLMs at Scale
Fixed state
Also written: state per conversation
A fixed-size memory each conversation needs in models like Qwen3.5, whose non-attention layers don't grow with length. It doesn't shrink when the KV cache is quantized.
Glossary: Serving LLMs at Scale
Also written: state per conversation
A fixed-size memory each conversation needs in models like Qwen3.5, whose non-attention layers don't grow with length. It doesn't shrink when the KV cache is quantized.