Entropy and Cross-Entropy
Problem statement
Implement entropy_and_cross_entropy(P, Q) for two discrete probability distributions given as equal-length lists. Return the tuple (H(P), H(P, Q)) as Python floats:
- Entropy $H(P)$ measures the uncertainty in a single distribution.
- Cross-entropy $H(P, Q)$ is the average code length (here in nats) needed to encode events drawn from $P$ using a code optimized for $Q$.
Use the natural logarithm. Terms with $P_i = 0$ contribute nothing to either quantity. If some outcome has $P_i > 0$ but $Q_i = 0$, the cross-entropy is infinite: return float('inf') for it.
Example
Input:
P = [0.7, 0.3], Q = [0.6, 0.4]
Output:
(0.610864, 0.632465)
Reasoning: Entropy: $H(P) = -(0.7\log(0.7) + 0.3\log(0.3)) \approx 0.611$. Cross-entropy: $H(P,Q) = -(0.7\log(0.6) + 0.3\log(0.4)) \approx 0.632$. Cross-entropy is higher because Q doesn't perfectly match P.
Show hint
Which distribution supplies the weights in each sum, and which one appears inside the logarithm?
Relevant roles: ML Engineer, Data Scientist, Deep Learning Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium