PvalProc macro

From MultXpert

Jump to: navigation, search

Return to Implementation of Multiple Testing Procedures.


The PvalProc macro computes multiplicity-adjusted p-values for the following nonparametric multiple testing procedures in general hypothesis testing problems with equally or unequally weighted null hypotheses:

  • Bonferroni Procedure.
  • Holm Procedure.
  • Hommel Procedure.
  • Hochberg Procedure.
  • Fixed-sequence Procedure.
  • Fallback Procedure.

General reference

Dmitrienko, A., Bretz, F., Westfall, P.H., Troendle, J., Wiens, B.L., Tamhane, A.C., Hsu, J.C. (2009). Multiple testing methodology. Multiple Testing Problems in Pharmaceutical Statistics. Dmitrienko, A., Tamhane, A.C., Bretz, F. (editors). Chapman and Hall/CRC Press, New York.


Download PvalProc macro.


Consider a clinical trial conducted to evaluate the efficacy of a novel treatment in patients with major depressive disorder. The trial evaluates the efficacy profile of three doses (labeled L, M and H) versus a placebo. Multiplicity in this problem is induced by the three dose-placebo tests and the null hypotheses of interest are defined as follows:

  • Null hypothesis H1: No difference between Dose H and Placebo.
  • Null hypothesis H2: No difference between Dose M and Placebo.
  • Null hypothesis H3: No difference between Dose L and Placebo.

Suppose that the one-sided p-values for the three dose-placebo tests, computed from an appropriate statistical model, e.g., an ANOVA model with an adjustment for baseline and a fixed term for the investigator, are given by:

  • Null hypothesis H1 (Dose H versus Placebo): p=0.0111.
  • Null hypothesis H2 (Dose M versus Placebo): p=0.0065.
  • Null hypothesis H3 (Dose L versus Placebo): p=0.0293.

The three tests and associated null hypotheses of no treatment effect are equally weighted.

To perform a multiplicity adjustment using a p-value-based multiple testing procedure, e.g., Bonferroni, Holm, Hommel, Hochberg, fixed-sequence or fallback procedure, the following data set needs to be defined:

   data example;
   input raw_p weight;
   0.0111 0.3333
   0.0065 0.3333
   0.0293 0.3334

Note that the ordering of p-values is important for multiple testing procedures with a pre-specified testing sequence, i.e., fixed-sequence and fallback procedures. These two multiple testing procedures are applied starting with the first p-value in the data set.

The EXAMPLE data set contains the raw p-values (RAW_P variable) as well as the hypothesis weights (WEIGHT variable). In this case the weights are equal to each other and thus the following shortcut can be used to define the EXAMPLE data set:

   data example;
   input raw_p;

The PvalProc macro is called as follows:


The first argument specifies the name of the data set with the raw p-values and hypothesis weights (IN=) and second argument defines the name of the data sets with multiplicity-adjusted p-values (OUT=). The PvalProc macro performs all six p-value-based multiple testing procedures and the user can select one or more multiple testing procedures:

   proc print data=adjp noobs label;
   format raw bonferroni holm hochberg hommel fixedseq fallback 6.4;
   var test raw bonferroni holm hochberg hommel fixedseq fallback;

The multiplicity-adjusted p-values generated by the macro are listed below:

   Test   Raw  Bonferroni  Holm    Hochberg  Hommel  sequence  Fallback
     1   0.0111   0.0333   0.0222   0.0222   0.0222   0.0111   0.0333
     2   0.0065   0.0195   0.0195   0.0195   0.0167   0.0111   0.0195
     3   0.0293   0.0879   0.0293   0.0293   0.0293   0.0293   0.0333

To perform inferences in the major depressive disorder trial, the adjusted p-values are compared to a one-sided α=0.025. For example, the Bonferroni procedure detects only one significant effect in the trial (Dose M versus Placebo, adjusted p=0.0195) and the Hommel procedure detects significant effect in two dose groups (Dose H versus Placebo, adjusted p=0.0222; Dose M versus Placebo, adjusted p=0.0167).

Unequally weighted tests

The multiplicity-adjusted p-values for the Bonferroni, Holm, Hommel and Hochberg procedures can also be computed using PROC MULTTEST:

   proc multtest pdata=example bonferroni holm hochberg hommel;

However, this SAS procedure does not support the fixed-sequence and fallback procedures and, secondly, PROC MULTTEST performs multiplicity adjustments only for hypothesis testing problems with equally weighted hypotheses.

The PvalProc macro computes multiplicity-adjusted p-values in general hypothesis testing problems with unequally weighted null hypotheses of no treatment effect. Suppose, for example, that the sponsor of the major depressive disorder trial expects the effect size at the highest dose (Dose H) to be greater than that at the two other doses (Doses L and M). The sponsor is planning to apply the fallback procedure with a greater weight assigned to the first test in the sequence to improve the overall probability of success. The following weights will be considered:

  • Null hypothesis H1 (Dose H versus Placebo): weight=0.50.
  • Null hypothesis H2 (Dose M versus Placebo): weight=0.25.
  • Null hypothesis H3 (Dose L versus Placebo): weight=0.25.

To implement the fallback procedure in this case, the EXAMPLE data sets needs to be defined as follows:

   data example;
   input raw_p weight;
   0.0111 0.50
   0.0065 0.25
   0.0293 0.25

and the PvalProc macro is invoked as before.

Other procedures

To find out more about software implementation of other multiple testing procedures and gatekeeping procedures, see Software.