Dirichlet-Multinomial Posterior and Prediction

Reported in interviews at Two Sigma

Problem statement

You have $K$ categories with unknown probabilities $\boldsymbol{\theta} = (\theta_1, \ldots, \theta_K)$, where $\boldsymbol{\theta}$ has a symmetric Dirichlet prior $\text{Dir}(\alpha_1, \ldots, \alpha_K)$ with all $\alpha_i = 1$ (i.e., a uniform prior over the simplex).

You observe $N$ draws from the multinomial, yielding counts $n_1, \ldots, n_K$ (so $\sum_i n_i = N$).

  1. What is the posterior distribution of $\boldsymbol{\theta}$ given the observed counts?
  1. Derive the posterior predictive probability that the next $m$ independent draws produce counts $c_1, \ldots, c_K$ (with $\sum_i c_i = m$).
  1. For a single next draw ($m = 1$), compute the predictive mean $E[X_i]$ and the predictive covariance $\text{Cov}(X_i, X_j)$, where $X_i$ is the indicator that the draw lands in category $i$.
Show hint

What does the product of the prior density and the likelihood look like when written as a function of the category probabilities?

Relevant roles: Data Scientist, Applied Scientist and ML Engineer.

Related free problems