Classic machine learning

Naive Bayes

Like sorting mail by combining clues from its words, Naive Bayes scores how well an example fits each class. It combines a class prior with feature likelihoods. Its shortcut assumes features are independent given the class, even when real data only roughly follows that assumption.

Observe two binary cluesBoth features are presentMultiply for class A0.4 x 0.5 x 0.5 = 0.10Multiply for class B0.6 x 0.1 x 0.2 = 0.012Normalize and choose A0.10 / 0.112 = 0.893
Each score is a prior times two class-conditional likelihoods; the model treats the clues as independent within each class.

How it works

  1. Estimate class priors from training labels or supply appropriate priors.
  2. Fit feature likelihoods within each class using a count, binary, or continuous variant.
  3. Apply smoothing to discrete likelihoods so unseen events do not force a zero score.
  4. Add log priors and log feature likelihoods to compute class scores.
  5. Choose the highest-scoring class or normalize scores to obtain model probabilities.

$$P(y=c\mid x)\propto P(y=c)\prod_jP(x_j\mid y=c)$$

The knobs that matter

Variant
Use multinomial for counts, Bernoulli for binary features, or Gaussian for continuous likelihoods.
Smoothing alpha
Larger values soften discrete likelihood estimates toward uniformity.
Class priors
Control the baseline probability assigned to each class.

What interviewers ask

  • What exactly is the independence assumption in Naive Bayes?
  • How do multinomial, Bernoulli and Gaussian Naive Bayes differ?
  • Why do you need smoothing and log probabilities?
  • Why can it classify well even when its independence assumption is false?

Common mistakes

  • Assuming features must be independent overall rather than conditional on the class.
  • Multiplying many tiny likelihoods directly and causing numerical underflow.
  • Treating the resulting probabilities as well calibrated without checking them.

When to use it

Use it for a fast baseline, especially with sparse word counts or binary text features. Strongly dependent features can double-count evidence and distort probabilities.

Implement it (3 problems)

Learn it

Interview questions about naive bayes