Skip to content

Glossary: Serving LLMs at Scale

Cache-ram

Also written: --cache-ram, --cache-idle-slots

llama-server options that keep a prompt cache in RAM, which a slot can restore from. Idle slots can be saved into it.

Where the book teaches it

Related terms

Prefix cachingSlot