Correcting Probabilities After Downsampling the Majority Class
Problem statement
Positives are rare, so you keep every positive and a random fraction $r$ of the negatives before training. The classifier now sees a different prior, and its probabilities are inflated by a known amount. Implement the sampling, the correction, and the check.
downsample_negatives(y, rate, rng)
- Return a boolean keep-mask: every positive (
y == 1) is kept, and each negative is kept whenrng.random(len(y)) < rateat its position (draw the full vector once).
corrected_probability(p_sampled, rate)
- A model trained on the sample estimates $p_s = \dfrac{p}{p + (1 - p)\,r}$ where $p$ is the probability under the original distribution. Invert it: return $p = \dfrac{r\, p_s}{r\, p_s + 1 - p_s}$, elementwise.
logit_shift(rate)
- The same correction as an additive shift on the log-odds scale, $\log r$; correcting a logistic model means adding this to its intercept.
sampled_probability(p, rate)
- The forward map above, so the two can be checked against each other.
calibration_gap(p, y)
mean(p) - mean(y): positive when the probabilities are inflated relative to the observed positive rate.
Example
Input:
corrected_probability(np.array([0.5]), 0.1)
Output:
[0.0909]
Reasoning: a model trained with only one negative in ten kept says 0.5, i.e. odds of 1. In the original data the odds are ten times lower, 0.1, so the probability is $0.1 / 1.1 = 0.0909$.
Show hint
How does thinning only the negatives change the odds of a positive, as opposed to the probability?
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium