Higher criticism for detecting sparse heterogeneous mixtures
Top Cited Papers
Open Access
- 1 June 2004
- journal article
- Published by Institute of Mathematical Statistics in The Annals of Statistics
- Vol. 32 (3)
- https://doi.org/10.1214/009053604000000265
Abstract
Higher criticism, or second-level significance testing, is a multiple-comparisons concept mentioned in passing by Tukey. It concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis. Tukey suggested comparing the fraction of observed significances at a given \alpha-level to the expected fraction under the joint null. In fact, he suggested standardizing the difference of the two quantities and forming a z-score; the resulting z-score tests the significance of the body of significance tests. We consider a generalization, where we maximize this z-score over a range of significance levels 0<\alpha\leq\alpha_0. We are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem: testing whether n normal means are all zero versus the alternative that a small fraction is nonzero. The subtlety of this ``sparse normal means'' testing problem can be seen from work of Ingster and Jin, who studied such problems in great detail. In their studies, they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p-values either for the bulk of the tests or for the few most highly significant tests. In this range, when the amplitude of nonzero means is calibrated with the fraction of nonzero means, the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses.Comment: Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Statistics (http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/00905360400000026This publication has 17 references indexed in Scilit:
- Adapting to unknown sparsity by controlling the false discovery rateThe Annals of Statistics, 2006
- Minimax nonparametric hypothesis testing for ellipsoids and Besov bodiesESAIM: Probability and Statistics, 2000
- Alignments in two-dimensional random sets of pointsAdvances in Applied Probability, 1980
- The Asymptotic Distribution of the Suprema of the Standardized Empirical ProcessesThe Annals of Statistics, 1979
- The Asymptotic Distribution of the Supremum of the Standardized Empirical Distribution Function on SubintervalsThe Annals of Statistics, 1979
- Limit theorems for the ratio of the empirical distribution function to the true distribution functionProbability Theory and Related Fields, 1978
- On Asymptotically Optimal Non-Parametric CriteriaTheory of Probability and Its Applications, 1968
- A limit theorem for the maximum of normalized sums of independent random variablesDuke Mathematical Journal, 1956
- Reliable and questionable significance in a series of statistical tests.Psychological Bulletin, 1952
- Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic ProcessesThe Annals of Mathematical Statistics, 1952