Bayes rules

Let ${\displaystyle \pi (\theta )\,\!}$ be a probability distribution on the states of nature. From a Bayesian point of view, we would regard it as a prior distribution. That is, it is our believed probability distribution on the states of nature, prior to observing data. For a frequentist, it is merely a function on ${\displaystyle \Theta \,\!}$ with no such special interpretation. The Bayes risk of the decision rule ${\displaystyle \delta \,\!}$ with respect to ${\displaystyle \pi (\theta )\,\!}$ is the expectation

${\displaystyle r(\pi ,\delta )=\operatorname {E} _{\pi (\theta )}[R(\theta ,\delta )].\,\!}$

A decision rule ${\displaystyle \delta \,\!}$ that minimizes ${\displaystyle r(\pi ,\delta )\,\!}$ is called a Bayes rule with respect to ${\displaystyle \pi (\theta )\,\!}$. There may be more than one such Bayes rule. If the Bayes risk is infinite for all ${\displaystyle \delta \,\!}$, then no Bayes rule is defined.

Generalized Bayes rules

In the Bayesian approach to decision theory, the observed ${\displaystyle x\,\!}$ is considered fixed. Whereas the frequentist approach (i.e., risk) averages over possible samples ${\displaystyle x\in {\mathcal {X}}\,\!}$, the Bayesian would fix the observed sample ${\displaystyle x\,\!}$ and average over hypotheses ${\displaystyle \theta \in \Theta \,\!}$. Thus, the Bayesian approach is to consider for our observed ${\displaystyle x\,\!}$ the expected loss

${\displaystyle \rho (\pi ,\delta \mid x)=\operatorname {E} _{\pi (\theta \mid x)}[L(\theta ,\delta (x))].\,\!}$

where the expectation is over the posterior of ${\displaystyle \theta \,\!}$ given ${\displaystyle x\,\!}$ (obtained from ${\displaystyle \pi (\theta )\,\!}$ and ${\displaystyle F(x\mid \theta )\,\!}$ using Bayes' theorem).

Having made explicit the expected loss for each given ${\displaystyle x\,\!}$ separately, we can define a decision rule ${\displaystyle \delta \,\!}$ by specifying for each ${\displaystyle x\,\!}$ an action ${\displaystyle \delta (x)\,\!}$ that minimizes the expected loss. This is known as a generalized Bayes rule with respect to ${\displaystyle \pi (\theta )\,\!}$. There may be more than one generalized Bayes rule, since there may be multiple choices of ${\displaystyle \delta (x)\,\!}$ that achieve the same expected loss.

At first, this may appear rather different from the Bayes rule approach of the previous section, not a generalization. However, notice that the Bayes risk already averages over ${\displaystyle \Theta \,\!}$ in Bayesian fashion, and the Bayes risk may be recovered as the expectation over ${\displaystyle {\mathcal {X}}}$ of the expected loss (where ${\displaystyle x\sim \theta \,\!}$ and ${\displaystyle \theta \sim \pi \,\!}$). Roughly speaking, ${\displaystyle \delta \,\!}$ minimizes this expectation of expected loss (i.e., is a Bayes rule) if and only if it minimizes the expected loss for each ${\displaystyle x\in {\mathcal {X}}}$ separately (i.e., is a generalized Bayes rule).

Then why is the notion of generalized Bayes rule an improvement? It is indeed equivalent to the notion of Bayes rule when a Bayes rule exists and all ${\displaystyle x\,\!}$ have positive probability. However, no Bayes rule exists if the Bayes risk is infinite (for all ${\displaystyle \delta \,\!}$). In this case it is still useful to define a generalized Bayes rule ${\displaystyle \delta \,\!}$, which at least chooses a minimum-expected-loss action ${\displaystyle \delta (x)\!\,}$ for those ${\displaystyle x\,\!}$ for which a finite-expected-loss action does exist. In addition, a generalized Bayes rule may be desirable because it must choose a minimum-expected-loss action ${\displaystyle \delta (x)\,\!}$ for every ${\displaystyle x\,\!}$, whereas a Bayes rule would be allowed to deviate from this policy on a set ${\displaystyle X\subseteq {\mathcal {X}}}$ of measure 0 without affecting the Bayes risk.

More important, it is sometimes convenient to use an improper prior ${\displaystyle \pi (\theta )\,\!}$. In this case, the Bayes risk is not even well-defined, nor is there any well-defined distribution over ${\displaystyle x\,\!}$. However, the posterior ${\displaystyle \pi (\theta \mid x)\,\!}$—and hence the expected loss—may be well-defined for each ${\displaystyle x\,\!}$, so that it is still possible to define a generalized Bayes rule.

Admissibility of (generalized) Bayes rules

According to the complete class theorems, under mild conditions every admissible rule is a (generalized) Bayes rule (with respect to some prior ${\displaystyle \pi (\theta )\,\!}$—possibly an improper one—that favors distributions ${\displaystyle \theta \,\!}$ where that rule achieves low risk). Thus, in frequentist decision theory it is sufficient to consider only (generalized) Bayes rules.

Conversely, while Bayes rules with respect to proper priors are virtually always admissible, generalized Bayes rules corresponding to improper priors need not yield admissible procedures. Stein's example is one such famous situation.