Glossary: Serving LLMs at Scale
Cache-ram
Also written: --cache-ram, --cache-idle-slots
llama-server options that keep a prompt cache in RAM, which a slot can restore from. Idle slots can be saved into it.
Glossary: Serving LLMs at Scale
Also written: --cache-ram, --cache-idle-slots
llama-server options that keep a prompt cache in RAM, which a slot can restore from. Idle slots can be saved into it.