Merge LoRA Adapter Weights for Zero-Latency Inference
Problem statement
Low-Rank Adaptation (LoRA) trains two small matrices A (shape r x k) and B (shape d x r) while keeping a pretrained weight matrix W0 (shape d x k) frozen. During training the effective weight is W0 + B*A, but at inference time we can fold the adapter directly into the base weights to eliminate any extra computation.
Implement a function merge_lora that takes the frozen weights W0, the LoRA matrices A and B, a scaling factor alpha, the rank r, and a boolean merge. When merge=True, return the merged weight matrix used at inference. When merge=False, treat W0 as a previously merged weight and return the recovered original (unmerged) base weights.
The LoRA update is scaled by alpha/r before being added.
Example
Input:
W0=[[1,2],[3,4]], A=[[1,0]], B=[[1],[1]], alpha=2, r=1, merge=True
Output:
[[3.0, 2.0], [5.0, 4.0]]
Reasoning: scaling = alpha/r = 2. B@A = [[1,0],[1,0]]. Delta = 2 * [[1,0],[1,0]] = [[2,0],[2,0]]. Merged = W0 + Delta = [[3,2],[5,4]].
Show hint
How is the effective weight of a LoRA layer written in terms of the frozen weight and the two small factors?
Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium