Model Memory Footprint Estimation
Problem statement
When training or deploying large neural networks, we need to estimate how much memory a model will use to pick hardware and avoid out-of-memory errors. The total depends on the parameters and also on whether gradients and optimizer states must be stored during training.
Implement estimate_model_memory(num_params, bytes_per_param, optimizer, include_gradients, unit) that computes the total memory footprint.
Rules
- Parameter memory is always counted:
num_params * bytes_per_parambytes. - If
include_gradientsis True, add anothernum_params * bytes_per_parambytes for gradients (same precision as parameters). - Optimizer states are stored in float32 (4 bytes per value):
'none': 0 extra bytes'sgd':num_params * 4bytes (one momentum buffer)'adam':num_params * 8bytes (first and second moment buffers)
- Convert the total bytes to
unit:'MB'divides by1024**2and'GB'by1024**3. - Return the result rounded to 2 decimal places.
Example
Input:
num_params = 124000000, bytes_per_param = 4, optimizer = 'none', include_gradients = False, unit = 'MB'
Output:
473.02
Reasoning: only parameters are stored, 124,000,000 * 4 = 496,000,000 bytes. Dividing by 1024^2 gives 473.0225 MB, which rounds to 473.02.
Show hint
Which separate buffers does a model need in memory during training, and which of them disappear at inference?
Relevant roles: ML Engineer, MLOps Engineer, LLM Engineer and AI Architect.
Related free problems
- Multi-Head AttentionMedium