LoRA-Adapted Linear Projection with Task Switching
Problem statement
Implement a class that wraps a frozen weight matrix with a trainable low-rank update, mimicking the kind of adapter commonly attached to attention projection layers in large transformers.
You are given:
- A frozen base weight matrix W0 of shape (d_out, d_in)
- Two trainable low-rank factors: A of shape (r, d_in) and B of shape (d_out, r)
- A scalar alpha used to scale the low-rank update
The effective (adapted) weight is W0 plus a scaled product of B and A. Your class must support:
- forward(x): apply the adapted linear map to an input vector x of shape (d_in,) and return a vector of shape (d_out,).
- num_trainable_params(): return the total number of trainable scalar parameters (only those in the low-rank factors, since W0 is frozen).
- merged_weight(): return the single dense matrix that is mathematically equivalent to the adapted layer (so it can be used at inference time with no extra cost).
- switch_task(A_new, B_new): replace the current low-rank factors with a new pair (e.g. to switch to another downstream task) while keeping W0 unchanged.
The scaling applied to the low-rank update should be alpha divided by r.
When printing arrays in tests, convert them to Python lists first.
The rank r is the number of rows of A. After switch_task, r (and therefore the scale alpha / r) comes from the new A, which may have a different rank from the old one.
Example
Input:
W0 = np.eye(2), A = [[1, 0]], B = [[1], [0]], alpha = 2, x = [1, 1]
Output:
[3.0, 1.0]
Reasoning: Rank r = 1, so the scale is alpha/r = 2. The low-rank update is scale B @ A = 2 [[1, 0], [0, 0]] = [[2, 0], [0, 0]]. Added to W0 = I, the adapted weight is [[3, 0], [0, 1]]. Multiplying by x = [1, 1] gives [3, 1].
Show hint
What is the effective weight of the layer in terms of the frozen matrix, the two factors and the scaling?
Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium