The Dirichlet process (DP) is a stochastic process that serves as a probability distribution over probability measures. It is a cornerstone of Bayesian nonparametric statistics, allowing models with an unbounded number of parameters.
Definition
Let $(\Theta,\mathcal{B})$ be a measurable space, $H$ a probability measure on $(\Theta,\mathcal{B})$ (called the base measure or base distribution), and $\alpha > 0$ a positive scalar (the concentration parameter). A random probability measure $G$ on $(\Theta,\mathcal{B})$ is said to follow a Dirichlet process with base measure $H$ and concentration $\alpha$, denoted $G \sim \mathrm{DP}(\alpha, H)$, if for every finite measurable partition $(A_1,\dots,A_k)$ of $\Theta$,
$$ (G(A_1),\dots,G(A_k)) \sim \operatorname{Dirichlet}\bigl(\alpha H(A_1),\dots,\alpha H(A_k)\bigr), $$
where $\operatorname{Dirichlet}(\cdot)$ denotes the finite‑dimensional Dirichlet distribution.
Key properties
- Discrete almost surely – A draw $G$ from a DP is a discrete probability measure with countably infinite atoms, even when the base measure $H$ is continuous.
- Expectation – $\mathbb{E}[G(A)] = H(A)$ for any measurable set $A$. The base measure thus represents the prior mean of the random measure.
- Concentration – Larger $\alpha$ yields draws that are more tightly concentrated around $H$; as $\alpha \to 0$, draws become increasingly spiky, concentrating mass on fewer atoms.
- Posterior conjugacy – If observations $X_1,\dots,X_n$ are i.i.d. from $G$ and the prior is $\mathrm{DP}(\alpha, H)$, the posterior distribution of $G$ is also a DP:
$$ G \mid X_{1:n} \sim \mathrm{DP}!\Bigl(\alpha + n,; \frac{\alpha}{\alpha + n}H + \frac{n}{\alpha + n}\widehat{H}_n\Bigr), $$
where $\widehat{H}_n$ is the empirical distribution of the observed data.
Constructive representations
- Stick‑breaking construction (Sethuraman, 1994):
$$ G = \sum_{k=1}^{\infty} \pi_k ,\delta_{\theta_k}, $$
with
$$
\pi_k = V_k \prod_{l=1}^{k-1}(1-V_l), \quad V_k \stackrel{\text{i.i.d.}}{\sim}\mathrm{Beta}(1,\alpha),\qquad
\theta_k \stackrel{\text{i.i.d.}}{\sim} H,
$$
and $\delta_{\theta_k}$ denoting a point mass at $\theta_k$.
- Chinese restaurant process (CRP) – The predictive distribution of a DP induces a partition structure known as the CRP, used for clustering in mixture models.
Applications
- Dirichlet process mixture models – Combine a DP prior over mixing measures with a parametric kernel (e.g., Gaussian) to obtain flexible, infinite‑component mixture models.
- Topic modeling – The hierarchical Dirichlet process (HDP) extends the DP to multiple groups, forming the basis of nonparametric extensions of latent Dirichlet allocation.
- Density estimation, survival analysis, topic evolution, and hierarchical clustering – Anywhere a flexible prior over unknown distributions is required.
Historical background
The Dirichlet process was introduced by Thomas S. Ferguson in his 1973 paper “A Bayesian Analysis of Some Nonparametric Problems”. It generalizes the finite‑dimensional Dirichlet distribution to the space of probability measures, building on earlier work on random probability measures and exchangeable partitions.
Related concepts
- Pitman–Yor process – A two‑parameter generalization of the DP that yields power‑law behavior in the sizes of atoms.
- Beta process – Conjugate prior for feature allocation models, analogous to the DP for clustering.
- Gaussian process – Another stochastic process used as a prior over functions, differing from the DP which is a prior over measures.
References
- Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1(2), 209–230.
- Sethuraman, J. (1994). A constructive definition of Dirichlet priors. Statistica Sinica, 4(2), 639–650.
- Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476), 1566–1581.
The Dirichlet process remains a fundamental tool for Bayesian modeling where the number of underlying components need not be fixed in advance.
