Entropy and Cross-Entropy

EasyML Coding

Problem statement

Implement entropy_and_cross_entropy(P, Q) for two discrete probability distributions given as equal-length lists. Return the tuple (H(P), H(P, Q)) as Python floats:

  • Entropy $H(P)$ measures the uncertainty in a single distribution.
  • Cross-entropy $H(P, Q)$ is the average code length (here in nats) needed to encode events drawn from $P$ using a code optimized for $Q$.

Use the natural logarithm. Terms with $P_i = 0$ contribute nothing to either quantity. If some outcome has $P_i > 0$ but $Q_i = 0$, the cross-entropy is infinite: return float('inf') for it.

Example

Input:

P = [0.7, 0.3], Q = [0.6, 0.4]

Output:

(0.610864, 0.632465)

Reasoning: Entropy: $H(P) = -(0.7\log(0.7) + 0.3\log(0.3)) \approx 0.611$. Cross-entropy: $H(P,Q) = -(0.7\log(0.6) + 0.3\log(0.4)) \approx 0.632$. Cross-entropy is higher because Q doesn't perfectly match P.

Show hint

Which distribution supplies the weights in each sum, and which one appears inside the logarithm?

Relevant roles: ML Engineer, Data Scientist, Deep Learning Engineer and Research Scientist.

Related free problems