Roofline Estimate: 70B Decoding Speed on One GPU

MediumLLMs & GenAI

Reported in interviews at NVIDIA Together AI

Problem statement

A team wants to serve a 70B-parameter dense decoder on a single 80 GB GPU with about 3.35 TB/s of memory bandwidth and about 1,979 TFLOP/s of dense FP8 compute (1 GB = $10^9$ bytes, 1 TB = $10^{12}$ bytes). The model has 80 layers and grouped-query attention with 8 KV heads of dimension 128. They care first about single-user decoding speed at batch size 1, then about throughput.

Show hint

Start by counting the bytes of weights at each precision.

Relevant roles: LLM Engineer, Deep Learning Engineer, MLOps Engineer and AI Architect.

Related free problems