Simple Self-Attention Without Trainable Weights
Problem statement
Implement a simplified self-attention mechanism that computes context vectors for each token in an input sequence, without any trainable weights.
Given an input matrix X of shape (T, d) where each row is a d-dimensional token embedding, compute context vectors Z of shape (T, d) as follows:
- For each query token i, compute attention scores by taking the dot product of x_i with every token x_j in the sequence (including itself). This yields a (T, T) score matrix.
- Normalize each row of the score matrix using softmax to obtain attention weights that sum to 1.
- Each context vector z_i is the weighted sum of all input tokens x_j, using the attention weights from row i.
Return the resulting context matrix as a list of lists, with each value rounded in the test (not in your solution).
Example
Input:
X = [[1.0, 0.0], [0.0, 1.0]]
Output:
[[0.7311, 0.2689], [0.2689, 0.7311]]
Reasoning: Scores matrix = X @ X.T = [[1, 0], [0, 1]]. Applying softmax row-wise: row 0 -> [e^1/(e^1+e^0), e^0/(e^1+e^0)] = [0.7311, 0.2689]. Context vector 0 = 0.7311[1,0] + 0.2689[0,1] = [0.7311, 0.2689]. By symmetry, context vector 1 = [0.2689, 0.7311].
Show hint
How do you measure how similar each token is to every other token using only the embeddings?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.
Related free problems
- Masked Self-AttentionMedium
- Gated AttentionMedium
- Multi-Head AttentionMedium