LoRA: Low-Rank Adaptation Forward Pass

EasyML Coding

Problem statement

Implement lora_forward(x, W, A, B, alpha=1.0) for LoRA (Low-Rank Adaptation), a parameter-efficient fine-tuning method. The pretrained weight $W$ is frozen, and the update is represented by two small trainable matrices whose product has rank at most $r$.

Shapes: input x is (batch, in_features), frozen W is (in_features, out_features), B is (in_features, r) and A is (r, out_features). The rank $r$ is the number of rows of A. Compute

$$\text{output} = x W + \frac{\alpha}{r} \, x B A$$

and return it as a list of lists.

Example

Input:

x = [[1.0, 2.0]], W = [[1.0, 0.0], [0.0, 1.0]], B = [[1.0], [1.0]], A = [[0.5, 0.5]], alpha = 2.0

Output:

[[4.0, 5.0]]

Reasoning: the frozen path gives $xW = [1, 2]$. The adapter path gives $xB = [3]$, then $xBA = [1.5, 1.5]$, scaled by $\alpha / r = 2$ to $[3, 3]$. The sum is $[4, 5]$.

Show hint

How can a weight update be written so that it has low rank and few trainable parameters?

Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer, AI Engineer and Deep Learning Engineer.

Related free problems