Simple Self-Attention Without Trainable Weights

EasyML Coding

Problem statement

Implement a simplified self-attention mechanism that computes context vectors for each token in an input sequence, without any trainable weights.

Given an input matrix X of shape (T, d) where each row is a d-dimensional token embedding, compute context vectors Z of shape (T, d) as follows:

  1. For each query token i, compute attention scores by taking the dot product of x_i with every token x_j in the sequence (including itself). This yields a (T, T) score matrix.
  2. Normalize each row of the score matrix using softmax to obtain attention weights that sum to 1.
  3. Each context vector z_i is the weighted sum of all input tokens x_j, using the attention weights from row i.

Return the resulting context matrix as a list of lists, with each value rounded in the test (not in your solution).

Example

Input:

X = [[1.0, 0.0], [0.0, 1.0]]

Output:

[[0.7311, 0.2689], [0.2689, 0.7311]]

Reasoning: Scores matrix = X @ X.T = [[1, 0], [0, 1]]. Applying softmax row-wise: row 0 -> [e^1/(e^1+e^0), e^0/(e^1+e^0)] = [0.7311, 0.2689]. Context vector 0 = 0.7311[1,0] + 0.2689[0,1] = [0.7311, 0.2689]. By symmetry, context vector 1 = [0.2689, 0.7311].

Show hint

How do you measure how similar each token is to every other token using only the embeddings?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.

Related free problems