What the KV Cache Stores and Why It Speeds Up Decoding
Problem statement
What does a language model cache when it generates text, and why? A team serves a decoder-only model with rotary position embeddings. Long conversations use substantial memory, so the team is considering fewer key and value heads to serve more conversations at once. Explain reuse during decoding and derive the cache memory requirements.
Show hint
Under causal attention, ask which quantities for past tokens can change when a new token arrives.
Relevant roles: LLM Engineer, Deep Learning Engineer, MLOps Engineer and AI Engineer.