Dirichlet-Multinomial Posterior and Prediction
Problem statement
You have $K$ categories with unknown probabilities $\boldsymbol{\theta} = (\theta_1, \ldots, \theta_K)$, where $\boldsymbol{\theta}$ has a symmetric Dirichlet prior $\text{Dir}(\alpha_1, \ldots, \alpha_K)$ with all $\alpha_i = 1$ (i.e., a uniform prior over the simplex).
You observe $N$ draws from the multinomial, yielding counts $n_1, \ldots, n_K$ (so $\sum_i n_i = N$).
- What is the posterior distribution of $\boldsymbol{\theta}$ given the observed counts?
- Derive the posterior predictive probability that the next $m$ independent draws produce counts $c_1, \ldots, c_K$ (with $\sum_i c_i = m$).
- For a single next draw ($m = 1$), compute the predictive mean $E[X_i]$ and the predictive covariance $\text{Cov}(X_i, X_j)$, where $X_i$ is the indicator that the draw lands in category $i$.
Show hint
What does the product of the prior density and the likelihood look like when written as a function of the category probabilities?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.