# Commonly Used Multiple Testing Procedures

### From MultXpert

This article lists key publications on multiple testing procedures used in multiplicity problems with a single family of objectives (or single family of null hypotheses of no effect). This includes

- Nonparametric procedures that do not make any assumptions about the joint distribution of the hypothesis test statistics.
- Semiparametric procedures based on decision rules that do not explicitly depend on the joint distribution of the hypothesis test statistics but additional distributional assumptions are needed in order to establish familywise error rate control.
- Parametric procedures that make explicit assumptions about the joint distribution of the hypothesis test statistics.

For information on publications dealing with gatekeeping procedures, see Gatekeeping Papers.

## Nonparametric procedures

**Holm procedure**

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics. 6, 65-70.

Strassburger, K., Bretz, F. (2008). Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni based closed tests. Statistics in Medicine. 27, 4914-4927.

Guilbaud, O. (2008). Simultaneous confidence regions corresponding to Holm's stepdown procedure and other closed-testing procedures. Biometrical Journal. 5, 678-692.

**Bonferroni-based chain procedures**

Hommel, G., Bretz, F., Maurer, W. (2007). Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Statistics in Medicine. 26, 4063-4073.

Bretz, F., Maurer, W., Brannath, W., Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine. 28, 586-604.

## Semiparametric procedures

**Simes global procedure**

Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika. 63, 655-660.

**Hochberg procedure**

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple significance testing. Biometrika. 75, 800-802.

Hochberg, Y., Rom, D. (1995). Extensions of multiple testing procedures based on Simesâ€™ test. Journal of Statistical Planning and Inference. 48, 141-152.

Tamhane, A.C., Liu, L. (2008). On weighted Hochberg procedures. Biometrika. 95, 279-294.

**Hommel procedure**

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika. 75, 383-386.

Hommel, G. (1989). A comparison of two modified Bonferroni procedures. Biometrika. 76, 624-625.

## Parametric procedures

**Dunnett family of parametric procedures**

Dunnett, C.W. (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association. 50, 1096-1121.

Dunnett, C.W., Tamhane, A.C. (1991). Step-down multiple tests for comparing treatments with a control in unbalanced one-way layouts. Statistics in Medicine. 10, 939-947.

Dunnett, C.W., Tamhane, A.C. (1992). A step-up multiple test procedure. Journal of the American Statistical Association. 87,162-170.

Dunnett, C.W., Tamhane, A.C. (1995). Step-up multiple testing of parameters with unequally correlated estimates. Biometrics. 51, 217-227.

**Parametric fallback and chain procedures**

Huque, M.F., Alosh, M. (2008). A flexible fixed-sequence testing method for hierarchically ordered correlated multiple endpoints in clinical trials. Journal of Statistical Planning and Inference. 138, 321-335.

Millen, B.A., Dmitrienko, A. (2011). Chain procedures: A class of flexible closed testing procedures with clinical trial applications. Statistics in Biopharmaceutical Research. 3, 14-30.

**Feedback and related procedures**

Alosh, M., Huque, M. (2009). A flexible strategy for testing subgroups and overall population. Statistics in Medicine. 28, 3-23.

Alosh, M., Huque, M. (2010). A consistency-adjusted alpha-adaptive strategy for sequential testing. Statistics in Medicine. 29, 1559-1571.

Li, J., Mehrotra, D.V. (2008). An efficient method for accommodating potentially underpowered primary endpoints. Statistics in Medicine. 27, 5377-5391.

Zhao, Y.D., Dmitrienko, A., Tamura, R. (2010). Design and analysis considerations in clinical trials with a sensitive subpopulation. Statistics in Biopharmaceutical Research. 2, 72-83.

## Other publications

For more information on multiple testing procedures in clinical trials, see

- Dmitrienko, A., Bretz, F., Westfall, P.H., Troendle, J., Wiens, B.L., Tamhane, A.C., Hsu, J.C. (2009). Multiple testing methodology. Multiple Testing Problems in Pharmaceutical Statistics. Dmitrienko, A., Tamhane, A.C., Bretz, F. (editors). Chapman and Hall/CRC Press, New York.