Optimal and maximin procedures for multiple testing problems

Abstract

Multiple testing problems (MTPs) are a staple of modern statistical analysis. The fundamental objective of MTPs is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family-wise error rate (FWER) or false discovery rate (FDR). In this paper we provide generalizations to MTPs of the optimal Neyman-Pearson test for a single hypothesis. We show that for simple hypotheses, for both FWER and FDR and relevant notions of power, finding the optimal multiple testing procedure can be formulated as infinite dimensional binary programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives that guarantee strong error control on the entire parameter space. We demonstrate the usefulness of these novel rules for identifying which studies contain signal in numerical experiments as well as in application to clinical trials with multiple studies. In various settings, the increase in power from using optimal and maximin procedures can range from 15% to more than 100%.