What the KV Cache Stores and Why It Speeds Up Decoding

MediumLLMs & GenAI

Reported in interviews at Amazon Apple Character.AI Mistral AI Netflix NVIDIA

Problem statement

What does a language model cache when it generates text, and why? A team serves a decoder-only model with rotary position embeddings. Long conversations use substantial memory, so the team is considering fewer key and value heads to serve more conversations at once. Explain reuse during decoding and derive the cache memory requirements.

Show hint

Under causal attention, ask which quantities for past tokens can change when a new token arrives.

Relevant roles: LLM Engineer, Deep Learning Engineer, MLOps Engineer and AI Engineer.

Related free problems