A new measure of the effective number of tests, a practical tool for comparing families of non‐independent significance tests

Abstract

p-Values from tests of significance can be combined using the Šidák correction (or the closely related Bonferroni correction) or Fisher's method, but both these methods require that the p-values combined be independent when all null hypotheses tested are true. In this paper adjustments to these methods are proposed, using a new eigenvalue-based measure of the effective number of independent tests to which the actual tests performed are equivalent, and are compared with adjustments proposed by previous authors. The adjusted methods are evaluated using a sample of 726 Alzheimer's disease (AD) cases and 707 group-matched controls, genotyped at 84,975 single-nucleotide polymorphism loci in 2,000 randomly chosen genes. The tests for genetic association with AD at loci within each gene are combined. The number of loci tested per gene varies from 2 to 994. The adjusted combined p-values agree well with the significance of the combined p-values determined empirically by random permutation of the data (Šidák correction: r=0.990; Fisher's method: r=0.994). This indicates that the combined p-values can be used to assess the relative strength of evidence for association of these genes with AD. The adjustment proposed here is a refinement of that of Nyholt ([2004] Am. J. Hum. Genet. 74:765–769), giving improved agreement with the results of random permutation. The improvement obtained is similar to that given by the refinement proposed by Li and Ji ([2005] Heredity 95:221–227). It is concluded that the concept of an effective number of tests is a valid approximation that allows p-values to be combined in a highly informative way. Genet. Epidemiol. 33:559–568, 2009.