Properties of higher criticism under strong dependence
Open Access
- 1 February 2008
- journal article
- research article
- Published by Institute of Mathematical Statistics in The Annals of Statistics
- Vol. 36 (1), 381-402
- https://doi.org/10.1214/009053607000000767
Abstract
The problem of signal detection using sparse, faint information is closely related to a variety of contemporary statistical problems, including the control of false-discovery rate, and classification using very high-dimensional data. Each problem can be solved by conducting a large number of simultaneous hypothesis tests, the properties of which are readily accessed under the assumption of independence. In this paper we address the case of dependent data, in the context of higher criticism methods for signal detection. Short-range dependence has no first-order impact on performance, but the situation changes dramatically under strong dependence. There, although higher criticism can continue to perform well, it can be bettered using methods based on differences of signal values or on the maximum of the data. The relatively inferior performance of higher criticism in such cases can be explained in terms of the fact that, under strong dependence, the higher criticism statistic behaves as though the data were partitioned into very large blocks, with all but a single representative of each block being eliminated from the dataset.Keywords
This publication has 20 references indexed in Scilit:
- Proportion of Non-Zero Normal Means: Universal Oracle Equivalences and Uniformly Consistent EstimatorsJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Theoretical Measures of Relative Performance of Classifiers for High Dimensional Data with Small Sample SizesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Goodness-of-fit tests via phi-divergencesThe Annals of Statistics, 2007
- Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple ComparisonsJournal of the American Statistical Association, 2007
- Asymptotic minimaxity of false discovery rate thresholding for sparse exponential dataThe Annals of Statistics, 2006
- Cosmological Non-Gaussian Signature Detection: Comparing Performance of Different Statistical TestsEURASIP Journal on Advances in Signal Processing, 2005
- A stochastic process approach to false discovery controlThe Annals of Statistics, 2004
- Detecting a target in very noisy data from multiple looksPublished by Institute of Mathematical Statistics ,2004
- Multiple Hypothesis Testing in Microarray ExperimentsStatistical Science, 2003
- Basic concepts of multiple tests — A surveyStatistical Papers, 2000