Vision Transformer (ViT) Image Encoder

MediumML Coding

Problem statement

Vision Transformers (ViTs) treat an image as a sequence of flattened patches and run them through a transformer encoder to get a global representation. A learnable classification (CLS) token is prepended to the patch sequence, and after the encoder its final embedding serves as the summary of the image (for example for a classification head).

Implement the forward pass of a minimal ViT encoder block that returns the final embedding of the CLS token: vit_encode(image, patch_size, W_patch, cls_token, pos_embed, W_qkv, W_o, num_heads), where

  • image has shape (H, W, C), with H and W divisible by patch_size.
  • patch_size is the side length of each square patch.
  • W_patch has shape (patch_size * patch_size * C, D) and linearly projects flattened patches to $D$-dimensional embeddings.
  • cls_token is a vector of shape (D,) placed at position 0 of the sequence.
  • pos_embed has shape (N+1, D), where $N$ is the number of patches, and is added element-wise to the token sequence.
  • W_qkv has shape (D, 3D) and produces concatenated queries, keys and values, in that order along the last axis.
  • W_o has shape (D, D) and is applied after multi-head attention.
  • num_heads divides D.

Take patches in row-major order and flatten each in C-order. Use scaled dot-product multi-head self-attention with scale $1/\sqrt{d_h}$, where $d_h = D / \text{num\_heads}$. Return the final embedding of the CLS token (position 0) as a Python list of length $D$.

Example

Input:

image = [[[1],[2]],[[3],[4]]]  # shape (2, 2, 1)
W_patch = [[1, 0]]
cls_token = [0, 0]
pos_embed = np.zeros((5, 2)).tolist()
W_qkv = [[1,0,1,0,1,0],[0,1,0,1,0,1]]  # Q = K = V = X
W_o = [[1,0],[0,1]]
print(vit_encode(image, 1, W_patch, cls_token, pos_embed, W_qkv, W_o, 1))

Output:

[2.0, 0.0]

Reasoning: the four 1-pixel patches embed to $[1,0], [2,0], [3,0], [4,0]$. Prepending the CLS token $[0,0]$ gives 5 tokens. The CLS query is zero, so every score is 0 and the attention weights are uniform (0.2 each). The output is the mean of the values, $[(0+1+2+3+4)/5, 0] = [2.0, 0.0]$, and W_o is the identity.

Show hint

Which steps turn an image into a token matrix, and where does the extra summary token sit in that sequence?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist, Applied Scientist and LLM Engineer.

Related free problems