Commonly Used Multiple Testing Procedures

From MultXpert

Jump to: navigation, search

This article lists key publications on multiple testing procedures used in multiplicity problems with a single family of objectives (or single family of null hypotheses of no effect). This includes

  • Nonparametric procedures that do not make any assumptions about the joint distribution of the hypothesis test statistics.
  • Semiparametric procedures based on decision rules that do not explicitly depend on the joint distribution of the hypothesis test statistics but additional distributional assumptions are needed in order to establish familywise error rate control.
  • Parametric procedures that make explicit assumptions about the joint distribution of the hypothesis test statistics.

For information on publications dealing with gatekeeping procedures, see Gatekeeping Papers.

Nonparametric procedures

Holm procedure

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics. 6, 65-70.

Strassburger, K., Bretz, F. (2008). Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni based closed tests. Statistics in Medicine. 27, 4914-4927.

Guilbaud, O. (2008). Simultaneous confidence regions corresponding to Holm's stepdown procedure and other closed-testing procedures. Biometrical Journal. 5, 678-692.

Bonferroni-based chain procedures

Hommel, G., Bretz, F., Maurer, W. (2007). Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Statistics in Medicine. 26, 4063-4073.

Bretz, F., Maurer, W., Brannath, W., Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine. 28, 586-604.

Semiparametric procedures

Simes global procedure

Simes, R.J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika. 63, 655-660.

Hochberg procedure

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple significance testing. Biometrika. 75, 800-802.

Hochberg, Y., Rom, D. (1995). Extensions of multiple testing procedures based on Simes’ test. Journal of Statistical Planning and Inference. 48, 141-152.

Tamhane, A.C., Liu, L. (2008). On weighted Hochberg procedures. Biometrika. 95, 279-294.

Hommel procedure

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika. 75, 383-386.

Hommel, G. (1989). A comparison of two modified Bonferroni procedures. Biometrika. 76, 624-625.

Parametric procedures

Dunnett family of parametric procedures

Dunnett, C.W. (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association. 50, 1096-1121.

Dunnett, C.W., Tamhane, A.C. (1991). Step-down multiple tests for comparing treatments with a control in unbalanced one-way layouts. Statistics in Medicine. 10, 939-947.

Dunnett, C.W., Tamhane, A.C. (1992). A step-up multiple test procedure. Journal of the American Statistical Association. 87,162-170.

Dunnett, C.W., Tamhane, A.C. (1995). Step-up multiple testing of parameters with unequally correlated estimates. Biometrics. 51, 217-227.

Parametric fallback and chain procedures

Huque, M.F., Alosh, M. (2008). A flexible fixed-sequence testing method for hierarchically ordered correlated multiple endpoints in clinical trials. Journal of Statistical Planning and Inference. 138, 321-335.

Millen, B.A., Dmitrienko, A. (2011). Chain procedures: A class of flexible closed testing procedures with clinical trial applications. Statistics in Biopharmaceutical Research. 3, 14-30.

Feedback and related procedures

Alosh, M., Huque, M. (2009). A flexible strategy for testing subgroups and overall population. Statistics in Medicine. 28, 3-23.

Alosh, M., Huque, M. (2010). A consistency-adjusted alpha-adaptive strategy for sequential testing. Statistics in Medicine. 29, 1559-1571.

Li, J., Mehrotra, D.V. (2008). An efficient method for accommodating potentially underpowered primary endpoints. Statistics in Medicine. 27, 5377-5391.

Zhao, Y.D., Dmitrienko, A., Tamura, R. (2010). Design and analysis considerations in clinical trials with a sensitive subpopulation. Statistics in Biopharmaceutical Research. 2, 72-83.

Other publications

For more information on multiple testing procedures in clinical trials, see

  • Dmitrienko, A., Bretz, F., Westfall, P.H., Troendle, J., Wiens, B.L., Tamhane, A.C., Hsu, J.C. (2009). Multiple testing methodology. Multiple Testing Problems in Pharmaceutical Statistics. Dmitrienko, A., Tamhane, A.C., Bretz, F. (editors). Chapman and Hall/CRC Press, New York.