Language models
Word embeddings
Think of a map where words with similar uses live in nearby neighborhoods. A word embedding assigns each word a learned vector of numbers. Training on nearby words makes those vectors useful for comparing meaning, though one fixed vector cannot capture every sense of a word.
How it works
- Collect center words and nearby context words from text.
- Look up a trainable vector for each center word and each context word.
- In skip-gram with negative sampling, raise scores for observed pairs and lower scores for sampled negative pairs.
- Use the learned word vectors as features or compare their directions with cosine similarity.
$$\operatorname{cos}(u,v)=\frac{u^\top v}{\lVert u\rVert\lVert v\rVert}$$
The knobs that matter
- embedding dimension
- more coordinates increase capacity and storage
- context window
- changes which nearby words count as training pairs
- negative samples
- more sampled negatives add work per observed pair
What interviewers ask
- How do word embeddings differ from bag-of-words features?
- How would you implement skip-gram with negative sampling?
- Why can two words have high cosine similarity, and when is that misleading?
Common mistakes
- Treating each coordinate as a named, human-readable feature.
- Assuming a static word vector changes when the word appears in a different sentence.
When to use it
Use word embeddings for compact lexical features and word similarity. Use contextual representations when the same word needs different meanings in different sentences.