In statistics, the Chapman–Robbins bound or Hammersley–Chapman–Robbins bound is a lower bound on the variance of estimators of a deterministic parameter. It is a generalization of the Cramér–Rao bound; compared to the Cramér–Rao bound, it is both tighter and applicable to a wider range of problems. However, it is usually more difficult to compute.

The bound was independently discovered by John Hammersley in 1950,[1] and by Douglas Chapman and Herbert Robbins in 1951.[2]

Statement

Let be the set of parameters for a family of probability distributions on .

For any two , let be the -divergence from to . Then:

Theorem  Given any scalar random variable , and any two , we have .

A generalization to the multivariable case is:[3]

Theorem  Given any multivariate random variable , and any ,

Proof

By the variational representation of chi-squared divergence:[3]

Plug in , to obtain:

Switch the denominator and the left side and take supremum over to obtain the single-variate case. For the multivariate case, we define for any . Then plug in in the variational representation to obtain:

Take supremum over , using the linear algebra fact that , we obtain the multivariate case.

Relation to Cramér–Rao bound

Usually, is the sample space of independent draws of a -valued random variable with distribution from a by parameterized family of probability distributions, is its -fold product measure, and is an estimator of . Then, for , the expression inside the supremum in the Chapman–Robbins bound converges to the Cramér–Rao bound of when , assuming the regularity conditions of the Cramér–Rao bound hold. This implies that, when both bounds exist, the Chapman–Robbins version is always at least as tight as the Cramér–Rao bound; in many cases, it is substantially tighter.

The Chapman–Robbins bound also holds under much weaker regularity conditions. For example, no assumption is made regarding differentiability of the probability density function p(x; θ) of . When p(x; θ) is non-differentiable, the Fisher information is not defined, and hence the Cramér–Rao bound does not exist.

See also

References

  1. Hammersley, J. M. (1950), "On estimating restricted parameters", Journal of the Royal Statistical Society, Series B, 12 (2): 192–240, JSTOR 2983981, MR 0040631
  2. Chapman, D. G.; Robbins, H. (1951), "Minimum variance estimation without regularity assumptions", Annals of Mathematical Statistics, 22 (4): 581–586, doi:10.1214/aoms/1177729548, JSTOR 2236927, MR 0044084
  3. 1 2 Polyanskiy, Yury (2017). "Lecture notes on information theory, chapter 29, ECE563 (UIUC)" (PDF). Lecture notes on information theory. Archived (PDF) from the original on 2022-05-24. Retrieved 2022-05-24.

Further reading

  • Lehmann, E. L.; Casella, G. (1998), Theory of Point Estimation (2nd ed.), Springer, pp. 113–114, ISBN 0-387-98502-6
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.