04 · How Models Run
KV Cache
Key-Value Cache
An inference optimization storing intermediate calculations so the model doesn't redo work for each new token. It's a major consumer of GPU memory and a key bottleneck in serving long conversations.
Concrete example
Why a very long chat can slow down or hit limits — the cache grows with every token.
Why it matters
Directly shapes cost and the practical limits of context length.