LoRA: Low-Rank Adaptation Forward Pass
Problem statement
Implement lora_forward(x, W, A, B, alpha=1.0) for LoRA (Low-Rank Adaptation), a parameter-efficient fine-tuning method. The pretrained weight $W$ is frozen, and the update is represented by two small trainable matrices whose product has rank at most $r$.
Shapes: input x is (batch, in_features), frozen W is (in_features, out_features), B is (in_features, r) and A is (r, out_features). The rank $r$ is the number of rows of A. Compute
$$\text{output} = x W + \frac{\alpha}{r} \, x B A$$
and return it as a list of lists.
Example
Input:
x = [[1.0, 2.0]], W = [[1.0, 0.0], [0.0, 1.0]], B = [[1.0], [1.0]], A = [[0.5, 0.5]], alpha = 2.0
Output:
[[4.0, 5.0]]
Reasoning: the frozen path gives $xW = [1, 2]$. The adapter path gives $xB = [3]$, then $xBA = [1.5, 1.5]$, scaled by $\alpha / r = 2$ to $[3, 3]$. The sum is $[4, 5]$.
Show hint
How can a weight update be written so that it has low rank and few trainable parameters?
Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer, AI Engineer and Deep Learning Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium