QLoRA: Quantized Low-Rank Adaptation Forward Pass
Problem statement
Implement the forward pass of QLoRA. The frozen pretrained weights are stored as 4-bit integers and dequantized to full precision during the forward pass, while the trainable LoRA matrices stay in full precision.
Inputs: x of shape (batch, in_features), integer quantized_W of shape (in_features, out_features) with a single scale and zero_point, B of shape (in_features, r) and A of shape (r, out_features), where $r$ is the number of rows of A.
- Dequantize: $W = \text{quantized\_W} \cdot \text{scale} + \text{zero\_point}$.
- Compute $\text{output} = x W + \frac{\alpha}{r} \, x B A$.
Return the output as a list of lists.
Example
Input:
x = [[1.0, 2.0]]
quantized_W = [[0, 10], [10, 0]]; scale = 0.1; zero_point = 0.0
B = [[0.5], [0.5]]; A = [[1.0, 1.0]]; alpha = 1.0
Output:
[[3.5, 2.5]]
Reasoning: dequantizing gives $W = [[0, 1], [1, 0]]$, so $xW = [2, 1]$. The adapter path is $xB = [1.5]$, $xBA = [1.5, 1.5]$, scaled by $\alpha / r = 1$. The sum is $[3.5, 2.5]$.
Show hint
How do you recover approximate real-valued weights from integers, a scale and a zero point?
Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer, AI Engineer and MLOps Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium