A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses
Top Cited Papers
Open Access
- 26 July 2013
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 8 (7), e69930
- https://doi.org/10.1371/journal.pone.0069930
Abstract
Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses. We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.Keywords
This publication has 28 references indexed in Scilit:
- Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic ReviewsInternational Journal of Epidemiology, 2012
- Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation studyStatistical Methods in Medical Research, 2010
- A simulation study comparing properties of heterogeneity measures in meta‐analysesStatistics in Medicine, 2006
- Simple Heterogeneity Variance Estimation for Meta-AnalysisJournal of the Royal Statistical Society Series C: Applied Statistics, 2005
- A simple confidence interval for meta‐analysisStatistics in Medicine, 2002
- Quantifying heterogeneity in a meta-analysisStatistics in Medicine, 2002
- Meta-analysis in clinical trialsControlled Clinical Trials, 1986
- Beta blockade during and after myocardial infarction: An overview of the randomized trialsProgress in Cardiovascular Diseases, 1985
- A random effects model for effect sizes.Psychological Bulletin, 1983
- Parametric Empirical Bayes Inference: Theory and ApplicationsJournal of the American Statistical Association, 1983