LoRA-Adapted Linear Projection with Task Switching

MediumML Coding

Problem statement

Implement a class that wraps a frozen weight matrix with a trainable low-rank update, mimicking the kind of adapter commonly attached to attention projection layers in large transformers.

You are given:

  • A frozen base weight matrix W0 of shape (d_out, d_in)
  • Two trainable low-rank factors: A of shape (r, d_in) and B of shape (d_out, r)
  • A scalar alpha used to scale the low-rank update

The effective (adapted) weight is W0 plus a scaled product of B and A. Your class must support:

  1. forward(x): apply the adapted linear map to an input vector x of shape (d_in,) and return a vector of shape (d_out,).
  2. num_trainable_params(): return the total number of trainable scalar parameters (only those in the low-rank factors, since W0 is frozen).
  3. merged_weight(): return the single dense matrix that is mathematically equivalent to the adapted layer (so it can be used at inference time with no extra cost).
  4. switch_task(A_new, B_new): replace the current low-rank factors with a new pair (e.g. to switch to another downstream task) while keeping W0 unchanged.

The scaling applied to the low-rank update should be alpha divided by r.

When printing arrays in tests, convert them to Python lists first.

The rank r is the number of rows of A. After switch_task, r (and therefore the scale alpha / r) comes from the new A, which may have a different rank from the old one.

Example

Input:

W0 = np.eye(2), A = [[1, 0]], B = [[1], [0]], alpha = 2, x = [1, 1]

Output:

[3.0, 1.0]

Reasoning: Rank r = 1, so the scale is alpha/r = 2. The low-rank update is scale B @ A = 2 [[1, 0], [0, 0]] = [[2, 0], [0, 0]]. Added to W0 = I, the adapted weight is [[3, 0], [0, 1]]. Multiplying by x = [1, 1] gives [3, 1].

Show hint

What is the effective weight of the layer in terms of the frozen matrix, the two factors and the scaling?

Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and AI Engineer.

Related free problems