Learned vs Sinusoidal Positional Encodings
Problem statement
How would you choose between learned and fixed positional encodings? A vision model turns image patches into tokens and adds learned position vectors. A text model uses the original transformer’s fixed sinusoidal encodings, and the team wants to understand the trade-offs. Explain relative offsets, compare the encodings, and adapt learned positions to larger images.
Show hint
Start from what self-attention does when you shuffle the input tokens.
Relevant roles: LLM Engineer, Deep Learning Engineer, Research Scientist and ML Engineer.
Related free problems
- Positional Encoding CalculatorML CodingMedium
- Rotary Positional Embeddings (RoPE)ML CodingMedium