Merge LoRA Adapter Weights for Zero-Latency Inference

EasyML Coding

Problem statement

Low-Rank Adaptation (LoRA) trains two small matrices A (shape r x k) and B (shape d x r) while keeping a pretrained weight matrix W0 (shape d x k) frozen. During training the effective weight is W0 + B*A, but at inference time we can fold the adapter directly into the base weights to eliminate any extra computation.

Implement a function merge_lora that takes the frozen weights W0, the LoRA matrices A and B, a scaling factor alpha, the rank r, and a boolean merge. When merge=True, return the merged weight matrix used at inference. When merge=False, treat W0 as a previously merged weight and return the recovered original (unmerged) base weights.

The LoRA update is scaled by alpha/r before being added.

Example

Input:

W0=[[1,2],[3,4]], A=[[1,0]], B=[[1],[1]], alpha=2, r=1, merge=True

Output:

[[3.0, 2.0], [5.0, 4.0]]

Reasoning: scaling = alpha/r = 2. B@A = [[1,0],[1,0]]. Delta = 2 * [[1,0],[1,0]] = [[2,0],[2,0]]. Merged = W0 + Delta = [[3,2],[5,4]].

Show hint

How is the effective weight of a LoRA layer written in terms of the frozen weight and the two small factors?

Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and AI Engineer.

Related free problems