Positional Encoding Calculator

MediumML Coding

Problem statement

Write a function pos_encoding(position, d_model) that computes the sinusoidal positional encodings used by the original Transformer, for a sequence of position positions and model dimension d_model.

For position $p$ and dimension index $j$ (0-based), let $i = \lfloor j / 2 \rfloor$ and

$$\theta_{p,j} = \frac{p}{10000^{2i / d_{model}}}, \qquad PE(p, j) = \begin{cases} \sin \theta_{p,j} & j \text{ even} \\ \cos \theta_{p,j} & j \text{ odd} \end{cases}$$

so sines and cosines are interleaved and each sine/cosine pair shares a frequency. Return a numpy array of shape (position, d_model) with dtype float16. If position is 0 or d_model is less than or equal to 0, return -1.

Example

Input:

print(pos_encoding(2, 8))

Output:

[[0.      1.      0.      1.      0.      1.      0.      1.     ]
 [0.8413  0.5405  0.09985 0.995   0.01    1.      0.001   1.     ]]

Reasoning: at position 0 every angle is 0, so the row alternates $\sin 0 = 0$ and $\cos 0 = 1$. At position 1 the angles are $1, 1, 0.1, 0.1, 0.01, 0.01, 0.001, 0.001$, giving $\sin 1 \approx 0.8413$, $\cos 1 \approx 0.5405$, and so on.

Show hint

How does the angle for a given position depend on the dimension index, and which dimensions are meant to share the same frequency?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Gen AI Engineer and Research Scientist.

Related free problems