In probability theory and statistics, the Dirichlet process (DP) is one of the most popular Bayesian nonparametric models. It was introduced by Thomas Ferguson^[1] as a prior over probability distributions.

A Dirichlet process ${\displaystyle \mathrm {DP} \left(s,G_{0}\right)}$ is completely defined by its parameters: ${\displaystyle G_{0}}$ (the base distribution or base measure) is an arbitrary distribution and ${\displaystyle s}$ (the concentration parameter) is a positive real number (it is often denoted as ${\displaystyle \alpha }$). According to the Bayesian paradigm these parameters should be chosen based on the available prior information on the domain.

The question is: how should we choose the prior parameters ${\displaystyle \left(s,G_{0}\right)}$ of the DP, in particular the infinite dimensional one ${\displaystyle G_{0}}$, in case of lack of prior information?

To address this issue, the only prior that has been proposed so far is the limiting DP obtained for ${\displaystyle s\rightarrow 0}$, which has been introduced under the name of Bayesian bootstrap by Rubin;^[2] in fact it can be proven that the Bayesian bootstrap is asymptotically equivalent to the frequentist bootstrap introduced by Bradley Efron.^[3] The limiting Dirichlet process ${\displaystyle s\rightarrow 0}$ has been criticized on diverse grounds. From an a-priori point of view, the main criticism is that taking ${\displaystyle s\rightarrow 0}$ is far from leading to a noninformative prior.^[4] Moreover, a-posteriori, it assigns zero probability to any set that does not include the observations.^[2]

The imprecise Dirichlet^[5] process has been proposed to overcome these issues. The basic idea is to fix ${\displaystyle s>0}$ but do not choose any precise base measure ${\displaystyle G_{0}}$.

More precisely, the imprecise Dirichlet process (IDP) is defined as follows:

${\displaystyle ~~\mathrm {IDP}$ :~\left\{\mathrm {DP} \left(s,G_{0}\right):~~G_{0}\in \mathbb {P} \right\}}

where ${\displaystyle \mathbb {P} }$ is the set of all probability measures. In other words, the IDP is the set of all Dirichlet processes (with a fixed ${\displaystyle s>0}$) obtained by letting the base measure ${\displaystyle G_{0}}$ to span the set of all probability measures.

Inferences with the Imprecise Dirichlet Process

Let ${\displaystyle P}$ a probability distribution on ${\displaystyle (\mathbb {X} ,{\mathcal {B}})}$ (here ${\displaystyle \mathbb {X} }$ is a standard Borel space with Borel ${\displaystyle \sigma }$-field ${\displaystyle {\mathcal {B}}}$) and assume that ${\displaystyle P\sim \mathrm {DP} (s,G_{0})}$. Then consider a real-valued bounded function ${\displaystyle f}$ defined on ${\displaystyle (\mathbb {X} ,{\mathcal {B}})}$. It is well known that the expectation of ${\displaystyle E[f]}$ with respect to the Dirichlet process is

${\displaystyle {\mathcal {E}}[E(f)]={\mathcal {E}}\left[\int f\,dP\right]=\int f\,d{\mathcal {E}}[P]=\int f\,dG_{0}.}$

One of the most remarkable properties of the DP priors is that the posterior distribution of ${\displaystyle P}$ is again a DP. Let ${\displaystyle X_{1},\dots ,X_{n}}$ be an independent and identically distributed sample from ${\displaystyle P}$ and ${\displaystyle P\sim Dp(s,G_{0})}$, then the posterior distribution of ${\displaystyle P}$ given the observations is

${\displaystyle P\mid X_{1},\dots ,X_{n}\sim Dp\left(s+n,G_{n}\right),~~~{\text{with}}~~~~~~G_{n}={\frac {s}{s+n}}G_{0}+{\frac {1}{s+n}}\sum \limits _{i=1}^{n}\delta _{X_{i}},}$

where ${\displaystyle \delta _{X_{i}}}$ is an atomic probability measure (Dirac's delta) centered at ${\displaystyle X_{i}}$. Hence, it follows that ${\displaystyle {\mathcal {E}}[E(f)\mid X_{1},\dots ,X_{n}]=\int f\,dG_{n}.}$ Therefore, for any fixed ${\displaystyle G_{0}}$, we can exploit the previous equations to derive prior and posterior expectations.

In the IDP ${\displaystyle G_{0}}$ can span the set of all distributions ${\displaystyle \mathbb {P} }$. This implies that we will get a different prior and posterior expectation of ${\displaystyle E(f)}$ for any choice of ${\displaystyle G_{0}}$. A way to characterize inferences for the IDP is by computing lower and upper bounds for the expectation of ${\displaystyle E(f)}$ w.r.t. ${\displaystyle G_{0}\in \mathbb {P} }$. A-priori these bounds are:

${\displaystyle {\underline {\mathcal {E}}}[E(f)]=\inf \limits _{G_{0}\in \mathbb {P} }\int f\,dG_{0}=\inf f,~~~~{\overline {\mathcal {E}}}[E(f)]=\sup \limits _{G_{0}\in \mathbb {P} }\int f\,dG_{0}=\sup f,}$

the lower (upper) bound is obtained by a probability measure that puts all the mass on the infimum (supremum) of ${\displaystyle f}$, i.e., ${\displaystyle G_{0}=\delta _{X_{0}}}$ with ${\displaystyle X_{0}=\arg \inf f}$ (or respectively with ${\displaystyle X_{0}=\arg \sup f}$). From the above expressions of the lower and upper bounds, it can be observed that the range of ${\displaystyle {\mathcal {E}}[E(f)]}$ under the IDP is the same as the original range of ${\displaystyle f}$. In other words, by specifying the IDP, we are not giving any prior information on the value of the expectation of ${\displaystyle f}$. A-priori, IDP is therefore a model of prior (near)-ignorance for ${\displaystyle E(f)}$.

A-posteriori, IDP can learn from data. The posterior lower and upper bounds for the expectation of ${\displaystyle E(f)}$ are in fact given by:

${\displaystyle {\begin{aligned}{\underline {\mathcal {E}}}[E(f)\mid X_{1},\dots ,X_{n}]&=\inf \limits _{G_{0}\in \mathbb {P} }\int f\,dG_{n}={\frac {s}{s+n}}\inf f+\int f(X){\frac {1}{s+n}}\sum \limits _{i=1}^{n}\delta _{X_{i}}(dX)\\&={\frac {s}{s+n}}\inf f+{\frac {n}{s+n}}{\frac {\sum \limits _{i=1}^{n}f(X_{i})}{n}},\\[6pt]{\overline {\mathcal {E}}}[E(f)\mid X_{1},\dots ,X_{n}]&=\sup \limits _{G_{0}\in \mathbb {P} }\int f\,dG_{n}={\frac {s}{s+n}}\sup f+\int f(X){\frac {1}{s+n}}\sum \limits _{i=1}^{n}\delta _{X_{i}}(dX)\\&={\frac {s}{s+n}}\sup f+{\frac {n}{s+n}}{\frac {\sum \limits _{i=1}^{n}f(X_{i})}{n}}.\end{aligned}}}$

It can be observed that the posterior inferences do not depend on ${\displaystyle G_{0}}$. To define the IDP, the modeler has only to choose ${\displaystyle s}$ (the concentration parameter). This explains the meaning of the adjective near in prior near-ignorance, because the IDP requires by the modeller the elicitation of a parameter. However, this is a simple elicitation problem for a nonparametric prior, since we only have to choose the value of a positive scalar (there are not infinitely many parameters left in the IDP model).

Finally, observe that for ${\displaystyle n\rightarrow \infty }$, IDP satisfies

${\displaystyle {\underline {\mathcal {E}}}\left[E(f)\mid X_{1},\dots ,X_{n}\right],\quad {\overline {\mathcal {E}}}\left[E(f)\mid X_{1},\dots ,X_{n}\right]\rightarrow S(f),}$

where ${\displaystyle S(f)=\lim _{n\rightarrow \infty }{\tfrac {1}{n}}\sum _{i=1}^{n}f(X_{i})}$. In other words, the IDP is consistent.

Lower (red) and Upper (blue) cumulative distribution for the observations {−1.17, 0.44, 1.17, 3.28, 1.44, 1.98}

Example: estimate of the cumulative distribution

Let ${\displaystyle X_{1},\dots ,X_{n}}$ be i.i.d. real random variables with cumulative distribution function ${\displaystyle F(x)}$.

Since ${\displaystyle F(x)=E[\mathbb {I} _{(\infty ,x]}]}$, where ${\displaystyle \mathbb {I} _{(\infty ,x]}}$ is the indicator function, we can use IDP to derive inferences about ${\displaystyle F(x).}$ The lower and upper posterior mean of ${\displaystyle F(x)}$ are

${\displaystyle {\begin{aligned}&{\underline {\mathcal {E}}}\left[F(x)\mid X_{1},\dots ,X_{n}\right]={\underline {\mathcal {E}}}[E(\mathbb {I} _{(\infty ,x]})\mid X_{1},\dots ,X_{n}]\\={}&{\frac {n}{s+n}}{\frac {\sum \limits _{i=1}^{n}\mathbb {I} _{(\infty ,x]}(X_{i})}{n}}={\frac {n}{s+n}}{\hat {F}}(x),\\[12pt]&{\overline {\mathcal {E}}}\left[F(x)\mid X_{1},\dots ,X_{n}\right]={\overline {\mathcal {E}}}\left[E(\mathbb {I} _{(\infty ,x]})\mid X_{1},\dots ,X_{n}\right]\\={}&{\frac {s}{s+n}}+{\frac {n}{s+n}}{\frac {\sum \limits _{i=1}^{n}\mathbb {I} _{(\infty ,x]}(X_{i})}{n}}={\frac {s}{s+n}}+{\frac {n}{s+n}}{\hat {F}}(x).\end{aligned}}}$

where ${\displaystyle {\hat {F}}(x)}$ is the empirical distribution function. Here, to obtain the lower we have exploited the fact that ${\displaystyle \inf \mathbb {I} _{(\infty ,x]}=0}$ and for the upper that ${\displaystyle \sup \mathbb {I} _{(\infty ,x]}=1}$.

Beta distributions for the lower (red) and upper (blue) probability corresponding to the observations {-1.17, 0.44, 1.17, 3.28, 1.44, 1.98}. The area in [0,0.5] gives the lower (0.891) and the upper (0.9375) probability of the hypothesis "the median is greater than zero".

Note that, for any precise choice of ${\displaystyle G_{0}}$ (e.g., normal distribution ${\displaystyle {\mathcal {N}}(x;0,1)}$), the posterior expectation of ${\displaystyle F(x)}$ will be included between the lower and upper bound.

Applications of the Imprecise Dirichlet Process

Dirichlet processes are frequently used in Bayesian nonparametric statistics. The Imprecise Dirichlet Process can be employed instead of the Dirichlet processes in any application in which prior information is lacking (it is therefore important to model this state of prior ignorance).

In this respect, the Imprecise Dirichlet Process has been used for nonparametric hypothesis testing, see the Imprecise Dirichlet Process statistical package. Based on the Imprecise Dirichlet Process, Bayesian nonparametric near-ignorance versions of the following classical nonparametric estimators have been derived: the Wilcoxon rank sum test^[5] and the Wilcoxon signed-rank test.^[6]

A Bayesian nonparametric near-ignorance model presents several advantages with respect to a traditional approach to hypothesis testing.

1. The Bayesian approach allows us to formulate the hypothesis test as a decision problem. This means that we can verify the evidence in favor of the null hypothesis and not only rejecting it and take decisions which minimize the expected loss.
2. Because of the nonparametric prior near-ignorance, IDP based tests allows us to start the hypothesis test with very weak prior assumptions, much in the direction of letting data speak for themselves.
3. Although the IDP test shares several similarities with a standard Bayesian approach, at the same time it embodies a significant change of paradigm when it comes to take decisions. In fact the IDP based tests have the advantage of producing an indeterminate outcome when the decision is prior-dependent. In other words, the IDP test suspends the judgment when the option which minimizes the expected loss changes depending on the Dirichlet Process base measure we focus on.
4. It has been empirically verified that when the IDP test is indeterminate, the frequentist tests are virtually behaving as random guessers. This surprising result has practical consequences in hypothesis testing. Assume that we are trying to compare the effects of two medical treatments (Y is better than X) and that, given the available data, the IDP test is indeterminate. In such a situation the frequentist test always issues a determinate response (for instance I can tell that Y is better than X), but it turns out that its response is completely random, like if we were tossing of a coin. On the other side, the IDP test acknowledges the impossibility of making a decision in these cases. Thus, by saying "I do not know", the IDP test provides a richer information to the analyst. The analyst could for instance use this information to collect more data.

Categorical variables

For categorical variables, i.e., when ${\displaystyle \mathbb {X} }$ has a finite number of elements, it is known that the Dirichlet process reduces to a Dirichlet distribution. In this case, the Imprecise Dirichlet Process reduces to the Imprecise Dirichlet model proposed by Walley^[7] as a model for prior (near)-ignorance for chances.

See also

Imprecise probability

Robust Bayesian analysis

References

External links

• Open source implementation of hypothesis tests based on the IDP
• The imprecise probability group at IDSIA