Learned vs Sinusoidal Positional Encodings

MediumLLMs & GenAI

Reported in interviews at Zomato

Problem statement

How would you choose between learned and fixed positional encodings? A vision model turns image patches into tokens and adds learned position vectors. A text model uses the original transformer’s fixed sinusoidal encodings, and the team wants to understand the trade-offs. Explain relative offsets, compare the encodings, and adapt learned positions to larger images.

Show hint

Start from what self-attention does when you shuffle the input tokens.

Relevant roles: LLM Engineer, Deep Learning Engineer, Research Scientist and ML Engineer.

Related free problems