Classic machine learning

Gaussian mixture model

Imagine sorting apples from two orchards whose fruit sizes overlap. Instead of assigning each apple to one orchard, you give each origin a probability based on size. A Gaussian mixture model does this with bell-shaped components, and expectation-maximization (EM) learns their centers, spreads and proportions.

-3-2-101234500.10.20.30.4xDensity
Component AComponent BMixture
Equal-weight Gaussians with means 0 and 2 and variance 1 overlap; at x = 1, each responsibility is 0.5.

How it works

  1. Choose the number of Gaussian components and initialize their means, covariances and mixing weights.
  2. In the E-step, compute each component’s probability of having generated each point, called its responsibility.
  3. In the M-step, use those responsibilities to update weighted means, covariances and mixing weights.
  4. Repeat until the log likelihood stops improving enough, and compare several initializations.

$$p(x)=\sum_{k=1}^{K}\pi_k\,\mathcal{N}(x;\mu_k,\Sigma_k)$$

The knobs that matter

n_components
more components fit finer groups but add parameters
covariance_type
full covariance allows tilted groups; diagonal uses fewer parameters
reg_covar
adds a small diagonal term to prevent singular covariances
n_init
more restarts reduce dependence on initialization

What interviewers ask

  • How does a Gaussian mixture model cluster data differently from k-means?
  • Can you walk me through the E-step and M-step?
  • How would you choose the number of components?
  • How would you use a mixture model to flag unusual transactions?

Common mistakes

  • Treating responsibilities as hard labels throws away uncertainty.
  • Without covariance regularization, a component can collapse onto a point and make the likelihood unbounded.
  • EM can reach a local optimum, so one initialization is not enough to judge the fit.

When to use it

Use it for soft clustering or density estimation when a few Gaussian components reasonably describe the data. Avoid interpreting low density as proof of fraud, or assuming every component represents a real category.

Implement it (3 problems)

Learn it

Interview questions about gaussian mixture model