Correcting Probabilities After Downsampling the Majority Class

EasyML Coding

Reported in interviews at Newsbreak

Problem statement

Positives are rare, so you keep every positive and a random fraction $r$ of the negatives before training. The classifier now sees a different prior, and its probabilities are inflated by a known amount. Implement the sampling, the correction, and the check.

downsample_negatives(y, rate, rng)

  • Return a boolean keep-mask: every positive (y == 1) is kept, and each negative is kept when rng.random(len(y)) < rate at its position (draw the full vector once).

corrected_probability(p_sampled, rate)

  • A model trained on the sample estimates $p_s = \dfrac{p}{p + (1 - p)\,r}$ where $p$ is the probability under the original distribution. Invert it: return $p = \dfrac{r\, p_s}{r\, p_s + 1 - p_s}$, elementwise.

logit_shift(rate)

  • The same correction as an additive shift on the log-odds scale, $\log r$; correcting a logistic model means adding this to its intercept.

sampled_probability(p, rate)

  • The forward map above, so the two can be checked against each other.

calibration_gap(p, y)

  • mean(p) - mean(y): positive when the probabilities are inflated relative to the observed positive rate.

Example

Input:

corrected_probability(np.array([0.5]), 0.1)

Output:

[0.0909]

Reasoning: a model trained with only one negative in ten kept says 0.5, i.e. odds of 1. In the original data the odds are ten times lower, 0.1, so the probability is $0.1 / 1.1 = 0.0909$.

Show hint

How does thinning only the negatives change the odds of a positive, as opposed to the probability?

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems