How conservative is Fisher's exact test? A quantitative evaluation of the two‐sample comparative binomial trial
- 12 March 2008
- journal article
- research article
- Published by Wiley in Statistics in Medicine
- Vol. 27 (18), 3598-3611
- https://doi.org/10.1002/sim.3221
Abstract
The debate as to which statistical methodology is most appropriate for the analysis of the two‐sample comparative binomial trial has persisted for decades. Practitioners who favor the conditional methods of Fisher, Fisher's exact test (FET), claim that only experimental outcomes containing the same amount of information should be considered when performing analyses. Hence, the total number of successes should be fixed at its observed level in hypothetical repetitions of the experiment. Using conditional methods in clinical settings can pose interpretation difficulties, since results are derived using conditional sample spaces rather than the set of all possible outcomes. Perhaps more importantly from a clinical trial design perspective, this test can be too conservative, resulting in greater resource requirements and more subjects exposed to an experimental treatment. The actual significance level attained by FET (the size of the test) has not been reported in the statistical literature. Berger (J. R. Statist. Soc. D (The Statistician) 2001; 50:79–85) proposed assessing the conservativeness of conditional methods using p‐value confidence intervals. In this paper we develop a numerical algorithm that calculates the size of FET for sample sizes, n, up to 125 per group at the two‐sided significance level, α=0.05. Additionally, this numerical method is used to define new significance levels α*=α+ε, where ε is a small positive number, for each n, such that the size of the test is as close as possible to the pre‐specified α (0.05 for the current work) without exceeding it. Lastly, a sample size and power calculation example are presented, which demonstrates the statistical advantages of implementing the adjustment to FET (using α* instead of α) in the two‐sample comparative binomial trial. Copyright © 2008 John Wiley & Sons, Ltd.Keywords
This publication has 13 references indexed in Scilit:
- P Values Maximized Over a Confidence Set for the Nuisance ParameterJournal of the American Statistical Association, 1994
- Fixing the number of events in large comparative trials with low event rates: A binomial approachControlled Clinical Trials, 1993
- Resolving the conflict over fisher's exact testThe Canadian Journal of Statistics / La Revue Canadienne de Statistique, 1992
- Fisher's Exact TestJournal of the Royal Statistical Society Series A: Statistics in Society, 1992
- The test of homogeneity for 2 × 2 contingency tables: A review of and some personal opinions on the controversy.Psychological Bulletin, 1990
- Fisher's inexact test: probability too serious to be left to statisticiansJAMA, 1989
- Exact Unconditional Sample Sizes for the 2 × 2 Binomial TrialJournal of the Royal Statistical Society. Series A (General), 1985
- Test of Significance for 2 × 2 Contingency TablesJournal of the Royal Statistical Society. Series A (General), 1984
- A Nonrandomized Unconditional Test for Comparing Two Proportions in 2×2 Contingency TablesTechnometrics, 1977
- A Nonrandomized Unconditional Test for Comparing Two Proportions in 2×2 Contigency TablesTechnometrics, 1977