Memory Savings of LoRA vs Full Fine-Tuning with Adam

EasyML Coding

Problem statement

When fine-tuning a large Transformer model with the Adam optimizer, GPU memory is consumed not just by the model weights, but also by the gradients and the two optimizer state tensors (the first and second moment estimates) that Adam maintains for every trainable parameter. For very large models this quadruples the parameter-related memory footprint compared to just storing the weights.

A common parameter-efficient alternative is to freeze the pre-trained weights and only train small low-rank update matrices injected into selected linear layers. For a target linear layer with input dimension d_in and output dimension d_out, the low-rank update introduces two trainable matrices of shapes (d_in, r) and (r, d_out), where r is the chosen rank. Because the original weights are frozen, no gradient or optimizer state needs to be stored for them, only for the new low-rank matrices.

Implement a function lora_memory_footprint that, given:

  • num_params_total: total number of parameters in the pre-trained model (int)
  • lora_target_shapes: list of (d_in, d_out) tuples describing each linear layer that is adapted with a low-rank update
  • rank: the rank r used for every low-rank adapter (int)
  • bytes_per_element: bytes used per parameter / gradient / optimizer-state element (int, default 2 for FP16)

returns a dictionary describing the memory accounting in bytes. Specifically, the function should report:

  • full_ft_bytes: bytes required when fine-tuning every parameter with Adam (weights + gradients + the two Adam moment buffers).
  • lora_bytes: bytes required when only the low-rank adapters are trained. The pre-trained weights are still kept on the GPU but in a frozen form (no gradient, no optimizer state); the adapter parameters need the full Adam treatment.
  • lora_trainable_params: the total number of trainable parameters introduced by all low-rank adapters.

All values must be integers.

Example

Input:

num_params_total=1000, lora_target_shapes=[(10, 10)], rank=2, bytes_per_element=2

Output:

{'full_ft_bytes': 8000, 'lora_bytes': 2320, 'lora_trainable_params': 40}

Reasoning: Full fine-tuning stores weights, gradients, and Adam's two moment buffers for all 1000 parameters: 1000 4 2 = 8000 bytes. The single adapted (10, 10) layer introduces 2 (10 + 10) = 40 trainable low-rank parameters. LoRA must still keep the 1000 frozen weights resident (1000 2 = 2000 bytes) but only the 40 adapter parameters need the full Adam quartet (40 4 2 = 320 bytes), totaling 2320 bytes.

Show hint

For each trainable scalar under Adam, how many same-sized tensors live on the device during a step?

Relevant roles: LLM Engineer, ML Engineer, MLOps Engineer, Gen AI Engineer and AI Architect.

Related free problems