Language models

Word embeddings

Think of a map where words with similar uses live in nearby neighborhoods. A word embedding assigns each word a learned vector of numbers. Training on nearby words makes those vectors useful for comparing meaning, though one fixed vector cannot capture every sense of a word.

00.20.40.60.811.200.20.40.60.811.2coordinate 1coordinate 2cat (1, 0)dog (0.8, 0.2)bus (0, 1)
Toy vectors: cat and dog point in similar directions from the origin.

How it works

  1. Collect center words and nearby context words from text.
  2. Look up a trainable vector for each center word and each context word.
  3. In skip-gram with negative sampling, raise scores for observed pairs and lower scores for sampled negative pairs.
  4. Use the learned word vectors as features or compare their directions with cosine similarity.

$$\operatorname{cos}(u,v)=\frac{u^\top v}{\lVert u\rVert\lVert v\rVert}$$

The knobs that matter

embedding dimension
more coordinates increase capacity and storage
context window
changes which nearby words count as training pairs
negative samples
more sampled negatives add work per observed pair

What interviewers ask

  • How do word embeddings differ from bag-of-words features?
  • How would you implement skip-gram with negative sampling?
  • Why can two words have high cosine similarity, and when is that misleading?

Common mistakes

  • Treating each coordinate as a named, human-readable feature.
  • Assuming a static word vector changes when the word appears in a different sentence.

When to use it

Use word embeddings for compact lexical features and word similarity. Use contextual representations when the same word needs different meanings in different sentences.

Implement it (1 problems)

Learn it

Interview questions about word embeddings