A new nonparametric approach for multiplicity control:Optimal Subset procedures

SummaryA new approach for multiplicity control (Optimal Subset) is presented. This is based on the selection of the best subset of partial (univariate) hypotheses producing the minimal p-value. In this work, we show how to perform this new procedure in the permutation framework, choosing suitable combining functions and permutation strategies. The optimal subset approach can be very useful in exploratory studies because it performs a weak control for multiplicity which can be a valid alternative to the False Discovery Rate (FDR). A comparative simulation study and an application to neuroimaging real data shows that it is particularly useful in presence of a high number of hypotheses. We also show how stepwise regression may be a special case of Optimal Subset procedures and how to adjust the p-value of the selected model taking into account for the multiplicity arising from the possible different models selected by a stepwise regression.