Classic machine learning
Naive Bayes
Like sorting mail by combining clues from its words, Naive Bayes scores how well an example fits each class. It combines a class prior with feature likelihoods. Its shortcut assumes features are independent given the class, even when real data only roughly follows that assumption.
How it works
- Estimate class priors from training labels or supply appropriate priors.
- Fit feature likelihoods within each class using a count, binary, or continuous variant.
- Apply smoothing to discrete likelihoods so unseen events do not force a zero score.
- Add log priors and log feature likelihoods to compute class scores.
- Choose the highest-scoring class or normalize scores to obtain model probabilities.
$$P(y=c\mid x)\propto P(y=c)\prod_jP(x_j\mid y=c)$$
The knobs that matter
- Variant
- Use multinomial for counts, Bernoulli for binary features, or Gaussian for continuous likelihoods.
- Smoothing alpha
- Larger values soften discrete likelihood estimates toward uniformity.
- Class priors
- Control the baseline probability assigned to each class.
What interviewers ask
- What exactly is the independence assumption in Naive Bayes?
- How do multinomial, Bernoulli and Gaussian Naive Bayes differ?
- Why do you need smoothing and log probabilities?
- Why can it classify well even when its independence assumption is false?
Common mistakes
- Assuming features must be independent overall rather than conditional on the class.
- Multiplying many tiny likelihoods directly and causing numerical underflow.
- Treating the resulting probabilities as well calibrated without checking them.
When to use it
Use it for a fast baseline, especially with sparse word counts or binary text features. Strongly dependent features can double-count evidence and distort probabilities.