Multiple Comparisons Problem

The multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values. The major issue in any discussion of multiple-comparison procedures is the question of the probability of Type I errors. Most differences among alternative techniques result from different approaches to the question of how to control these errors. The problem is in part technical; but it is really much more a subjective question of how you want to define the error rate and how large you are willing to let the maximum possible error rate be.^[7] Dunnett's test are well known and widely used in multiple comparison procedure for simultaneously comparing, by interval estimation or hypothesis testing, all active treatments with a control when sampling from a distribution where the normality assumption is reasonable. Dunnett's test is designed to hold the family-wise error rate at or below ${\displaystyle \alpha }$ when performing multiple comparisons of treatment group with control.^[7]

Uses of Dunnett’s test

The original work on Multiple Comparisons problem was made by Tukey and Scheffé. Their method was a general one, which considered all kinds of pairwise comparisons.^[7] Tukey's and Scheffé's methods allow any number of comparisons among a set of sample means. On the other hand, Dunnett's test only compares one group with the others, addressing a special case of multiple comparisons problem — pairwise comparisons of multiple treatment groups with a single control group. In the general case, where we compare each of the pairs, we make ${\displaystyle k(k-1){\big /}2}$ comparisons (where k is the number of groups), but in the treatment vs. controls case we will make only ${\displaystyle (k-1)}$ comparisons. If in the case of treatment and control groups we were to use the more general Tukey's and Scheffé's methods, they can result in unnecessarily wide confidence intervals. Dunnett's test takes into consideration the special structure of comparing treatment against control, yielding narrower confidence intervals.^[5]
It is very common to use Dunnett's test in medical experiments, for example comparing blood count measurements on three groups of animals, one of which served as a control while the other two were treated with two different drugs. Another common use of this method is among agronomists: agronomists may want to study the effect of certain chemicals added to the soil on crop yield, so they will leave some plots untreated (control plots) and compare them to the plots where chemicals were added to the soil (treatment plots).

Assumptions

The analysis considers the case where the results of the experiment are numerical, and the experiment is performed to compare p treatments with a control group. The results can be summarized as a set of ${\displaystyle (p+1)}$ calculated means of the sets of observations, ${\displaystyle ({\bar {X_{0}}},...,{\bar {X_{p}}})}$, while ${\displaystyle ({\bar {X_{1}}},...,{\bar {X_{p}}})}$ are referring to the treatment and ${\displaystyle {\bar {X_{0}}}}$ is referring to the control set of observations, and ${\displaystyle s}$ is an independent estimate of the common standard deviation of all ${\displaystyle p+1}$ sets of observations. All ${\displaystyle {\bar {X_{i}}}}$ of the ${\displaystyle p+1}$ sets of observations are assumed to be independently and normally distributed with a common variance ${\displaystyle \sigma ^{2}}$ and means ${\displaystyle \mu _{i}}$. There is also an assumption that there is an available estimate ${\displaystyle s^{2}}$ for ${\displaystyle \sigma ^{2}}$.

Calculation

Dunnett's test's calculation is a procedure that is based on calculating confidence statements about the true or the expected values of the ${\displaystyle p}$ differences ${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}}$, thus the differences between treatment groups' mean and control group's mean. This procedure ensures that the probability of all ${\displaystyle p}$ statements ${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}}$ being simultaneously correct is equal to a specified value, ${\displaystyle P}$. When calculating one sided upper (or lower) Confidence interval for the true value of the difference between the mean of the treatment and the control group, ${\displaystyle P}$ constitutes the probability that this actual value will be less than the upper (or greater than the lower) limit of that interval. When calculating two-sided confidence interval, ${\displaystyle P}$ constitutes the probability that the true value will be between the upper and the lower limits.

First, we will denote the available N observations by ${\displaystyle X_{ij}}$ when ${\displaystyle i=1,...,p}$ and ${\displaystyle j=1,...,N_{i}}$ and estimate the common variance by, for example: ${\displaystyle s^{2}={\frac {\sum _{i=0}^{p}\sum _{j=1}^{N_{i}}(X_{ij}-{\bar {X_{i}}})^{2}}{n}}}$ when ${\displaystyle {\bar {X_{i}}}}$ is the mean of group ${\displaystyle i}$ and ${\displaystyle N_{i}}$ is the number of observations in group ${\displaystyle i}$, and ${\displaystyle n=\sum _{i=0}^{p}N_{i}-(p+1)}$ degrees of freedom. As mentioned before, we would like to obtain separate confidence limits for each of the differences ${\displaystyle m_{i}-m_{0},(i=1,...,p)}$ such that the probability that all ${\displaystyle p}$ confidence intervals will contain the corresponding ${\displaystyle m_{i}-m_{0}}$ is equal to ${\displaystyle P}$.

We will consider the general case where there are ${\displaystyle p}$ treatment groups and one control group. We will write:

${\displaystyle z_{i}={\cfrac {{\bar {X_{i}}}-{\bar {X_{0}}}-(m_{i}-m_{0})}{\sqrt {{\cfrac {1}{N_{i}}}+{\cfrac {1}{N_{0}}}}}}}$

${\displaystyle D_{i}={\cfrac {{\bar {X_{i}}}-{\bar {X_{0}}}-(m_{i}-m_{0})}{s{\sqrt {{\cfrac {1}{N_{i}}}+{\cfrac {1}{N_{0}}}}}}}}$

we will also write: ${\displaystyle D_{i}={\frac {z_{i}}{s}}}$, which follows the Student's t-statistic distribution with n degrees of freedom. The lower confidence limits with joint confidence coefficient ${\displaystyle P}$ for the ${\displaystyle p}$ treatment effects ${\displaystyle m_{i}-m_{0},(i=1,...,p)}$ will be given by:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}-d_{i}'s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

and the ${\displaystyle p}$ constants ${\displaystyle d_{i}'}$ are chosen so that ${\displaystyle Prob(t_{1}<d_{1}',...,t_{p}<d_{p}')=P}$. Similarly, the upper limits will be given by:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}+d_{i}'s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

For bounding ${\displaystyle m_{i}-m_{0}}$ in both directions, the following interval might be taken:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}\pm d_{i}''s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

when ${\displaystyle d_{i}''}$ are chosen to satisfy ${\displaystyle Prob(|t_{1}|<d_{1}',...,|t_{p}|<d_{p}')=P}$. The solution to those particular values of ${\displaystyle d_{i}''}$ for two sided test and ${\displaystyle d_{i}'}$ for one sided test is given in the tables.^[5] An updated table of critical values was published in 1964.^[6]

Breaking strength of fabric

The following example was adapted from one given by Villars.^[10] The data represent measurements on the breaking strength of fabric treated by three different chemical process compared with a standard method of manufacture.^[5]

Here, p=3 and N=3. The average variance is ${\displaystyle s^{2}=19}$, which is an estimate of the common variance of the four sets with (p+1)(N-1)=8 degrees of freedom. This can be calculated as follows:

${\displaystyle {\frac {55^{2}+47^{2}+48^{2}+55^{2}+...+41^{2}-3(50^{2}+61^{2}+52^{2}+45^{2})}{8}}={\frac {152}{8}}=19}$.

The standard deviation is ${\displaystyle s={\sqrt {19}}=4.36}$ and the estimated standard error of a difference between two means is ${\displaystyle s{\sqrt {\frac {2}{N}}}=4.36{\sqrt {\frac {2}{N}}}=3.56}$.

The quantity which must be added to and/or subtracted from the observed differences between the means to give their confidence limits has been called by Tukey an "allowance" and is given by ${\displaystyle A=ts{\sqrt {\frac {2}{N}}}}$, where t drawn from the Multivariate t-distribution, or can be obtained from Dunnett's Table 1 if one side limits are desired or from Dunnett's Table 2 if two-sided limits are wanted. For p=3 and d.f.=8, t=2.42 for one side limits and t=2.88 for two-sided limits for p=95%. Analogous values of t can be determined from the tables if p=99% confidence is required. For one-sided limits, the allowance is A=(2.42)(3.56)=9 and the experimenter can conclude that:

• The breaking strength using process 1 exceeds the standard by at least ${\displaystyle 61-50-9=2lbs.}$
• The breaking strength using process 2 exceeds the standard by at least ${\displaystyle 52-50-9=-7lbs}$.
• The breaking strength using process 3 exceeds the standard by at least ${\displaystyle 45-50-9=-14lbs}$.

The joint statement consisting of the above three conclusions has a confidence coefficient of 95%, i.e., in the long run, 95% of such joint statements will actually be correct. Upper limits for the three differences could be obtained in an analogous manner. For two-sided limits, the allowance is A=(2.88)(3.56)=11 and the experimenter can conclude that:

• The breaking strength using process 1 exceeds the standard by an amount between

${\displaystyle 61-50-11=0lbs.}$ and ${\displaystyle 61-50+11=22lbs.}$

• The breaking strength using process 2 exceeds the standard by an amount between

${\displaystyle 52-50-11=-9lbs}$ and ${\displaystyle 52-50+11=13lbs}$.

• The breaking strength using process 3 exceeds the standard by an amount between

${\displaystyle 45-50-11=-16lbs}$ and ${\displaystyle 45-50+11=6lbs}$. The joint confidence coefficient for these three statement is greater than 95%. (Due to an approximation made in computing Tables 2a and 2b, the tabulated values of t are somewhat larger than necessary so that the actual p's attained are slightly greater than 95 and 99%.No such approximation was made in computing Tables 1a and 1b).