Vision Transformer (ViT) Image Encoder
Problem statement
Vision Transformers (ViTs) treat an image as a sequence of flattened patches and run them through a transformer encoder to get a global representation. A learnable classification (CLS) token is prepended to the patch sequence, and after the encoder its final embedding serves as the summary of the image (for example for a classification head).
Implement the forward pass of a minimal ViT encoder block that returns the final embedding of the CLS token: vit_encode(image, patch_size, W_patch, cls_token, pos_embed, W_qkv, W_o, num_heads), where
imagehas shape(H, W, C), withHandWdivisible bypatch_size.patch_sizeis the side length of each square patch.W_patchhas shape(patch_size * patch_size * C, D)and linearly projects flattened patches to $D$-dimensional embeddings.cls_tokenis a vector of shape(D,)placed at position 0 of the sequence.pos_embedhas shape(N+1, D), where $N$ is the number of patches, and is added element-wise to the token sequence.W_qkvhas shape(D, 3D)and produces concatenated queries, keys and values, in that order along the last axis.W_ohas shape(D, D)and is applied after multi-head attention.num_headsdividesD.
Take patches in row-major order and flatten each in C-order. Use scaled dot-product multi-head self-attention with scale $1/\sqrt{d_h}$, where $d_h = D / \text{num\_heads}$. Return the final embedding of the CLS token (position 0) as a Python list of length $D$.
Example
Input:
image = [[[1],[2]],[[3],[4]]] # shape (2, 2, 1)
W_patch = [[1, 0]]
cls_token = [0, 0]
pos_embed = np.zeros((5, 2)).tolist()
W_qkv = [[1,0,1,0,1,0],[0,1,0,1,0,1]] # Q = K = V = X
W_o = [[1,0],[0,1]]
print(vit_encode(image, 1, W_patch, cls_token, pos_embed, W_qkv, W_o, 1))
Output:
[2.0, 0.0]
Reasoning: the four 1-pixel patches embed to $[1,0], [2,0], [3,0], [4,0]$. Prepending the CLS token $[0,0]$ gives 5 tokens. The CLS query is zero, so every score is 0 and the attention weights are uniform (0.2 each). The output is the mean of the values, $[(0+1+2+3+4)/5, 0] = [2.0, 0.0]$, and W_o is the identity.
Show hint
Which steps turn an image into a token matrix, and where does the extra summary token sit in that sequence?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist, Applied Scientist and LLM Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium