In statistical hypothesis testing, specifically multiple hypothesis testing, the q-value in the Storey-Tibshirani procedure provides a means to control the positive false discovery rate (pFDR).^[1] Just as the p-value gives the expected false positive rate obtained by rejecting the null hypothesis for any result with an equal or smaller p-value, the q-value gives the expected pFDR obtained by rejecting the null hypothesis for any result with an equal or smaller q-value.

Schematic representation of the Storey-Tibshirani procedure for correcting for multiple hypothesis testing, assuming correctly calculated p-values. y-axis is frequency.

History

In statistics, testing multiple hypotheses simultaneously using methods appropriate for testing single hypotheses tends to yield many false positives: the so-called multiple comparisons problem.^[2] For example, assume that one were to test 1,000 null hypotheses, all of which are true, and (as is conventional in single hypothesis testing) to reject null hypotheses with a significance level of 0.05; due to random chance, one would expect 5% of the results to appear significant (P < 0.05), yielding 50 false positives (rejections of the null hypothesis).^[3] Since the 1950s, statisticians had been developing methods for multiple comparisons that reduced the number of false positives, such as controlling the family-wise error rate (FWER) using the Bonferroni correction, but these methods also increased the number of false negatives (i.e. reduced the statistical power).^[2] In 1995, Yoav Benjamini and Yosef Hochberg proposed controlling the false discovery rate (FDR) as a more statistically powerful alternative to controlling the FWER in multiple hypothesis testing.^[2] The pFDR and the q-value were introduced by John D. Storey in 2002 in order to improve upon a limitation of the FDR, namely that the FDR is not defined when there are no positive results.^[1][4]

Definition

Let there be a null hypothesis ${\displaystyle H_{0}}$ and an alternative hypothesis ${\displaystyle H_{1}}$. Perform ${\displaystyle m}$ hypothesis tests; let the test statistics be i.i.d. random variables ${\displaystyle T_{1},\ldots ,T_{m}}$ such that ${\displaystyle T_{i}\mid D_{i}\sim (1-D_{i})\cdot F_{0}+D_{i}\cdot F_{1}}$. That is, if ${\displaystyle H_{0}}$ is true for test ${\displaystyle i}$ (${\displaystyle D_{i}=0}$), then ${\displaystyle T_{i}}$ follows the null distribution ${\displaystyle F_{0}}$; while if ${\displaystyle H_{1}}$ is true (${\displaystyle D_{i}=1}$), then ${\displaystyle T_{i}}$ follows the alternative distribution ${\displaystyle F_{1}}$. Let ${\displaystyle D_{i}\sim \operatorname {Bernoulli} (\pi _{1})}$, that is, for each test, ${\displaystyle H_{1}}$ is true with probability ${\displaystyle \pi _{1}}$ and ${\displaystyle H_{0}}$ is true with probability ${\displaystyle \pi _{0}=1-\pi _{1}}$. Denote the critical region (the values of ${\displaystyle T_{i}}$ for which ${\displaystyle H_{0}}$ is rejected) at significance level ${\displaystyle \alpha }$ by ${\displaystyle \Gamma _{\alpha }}$. Let an experiment yield a value ${\displaystyle t}$ for the test statistic. The q-value of ${\displaystyle t}$ is formally defined as

${\displaystyle \inf _{\{\Gamma _{\alpha }:t\in \Gamma _{\alpha }\}}\operatorname {pFDR} (\Gamma _{\alpha })}$

That is, the q-value is the infimum of the pFDR if ${\displaystyle H_{0}}$is rejected for test statistics with values ${\displaystyle \geq t}$. Equivalently, the q-value equals

${\displaystyle \inf _{\{\Gamma _{\alpha }:t\in \Gamma _{\alpha }\}}\Pr(H=0\mid T\in \Gamma _{\alpha })}$

which is the infimum of the probability that ${\displaystyle H_{0}}$ is true given that ${\displaystyle H_{0}}$ is rejected (the false discovery rate).^[1]

Relationship to the p-value

The p-value is defined as

${\displaystyle \inf _{\{\Gamma _{\alpha }:t\in \Gamma _{\alpha }\}}\Pr(T\in \Gamma _{\alpha }\mid D=0)}$

the infimum of the probability that ${\displaystyle H_{0}}$ is rejected given that ${\displaystyle H_{0}}$ is true (the false positive rate). Comparing the definitions of the p- and q-values, it can be seen that the q-value is the minimum posterior probability that ${\displaystyle H_{0}}$ is true.^[1]

Interpretation

The q-value can be interpreted as the false discovery rate (FDR): the proportion of false positives among all positive results. Given a set of test statistics and their associated q-values, rejecting the null hypothesis for all tests whose q-value is less than or equal to some threshold ${\displaystyle \alpha }$ ensures that the expected value of the false discovery rate is ${\displaystyle \alpha }$.^[5]

Applications

Implementations

Note: the following is an incomplete list.

References