Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis
Open Access
- 9 March 2020
- journal article
- review article
- Published by Springer Science and Business Media LLC in Systematic Reviews
- Vol. 9 (1), 1-8
- https://doi.org/10.1186/s13643-020-01305-w
Abstract
As systematic reviews’ limited coverage of the medical literature necessitates decision-making based on unsystematic review, we investigated a possible advantage of systematic review (aside from dataset size and systematic analysis): does systematic review avoid potential bias in sampling primary studies from high impact factor journals? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, readers who focus on higher-impact journals for their rapid literature reviews may introduce bias which could be mitigated by complete, systematic sampling. We randomly sampled Cochrane Library (20 July 2005) treatment reviews that measured mortality as a binary outcome, published in English or French, with at least five RCTs with one or more deaths. Our domain-based assessment of risk of bias included funding source, randomness of allocation sequence, blinding, and allocation concealment. The primary analysis employed logistic regression by a generalized linear model with a generalized estimating equation to estimate the association between various factors and publication in a journal with a high journal impact factor (JIF). From the 29 included systematic reviews, 189 RCTs contributed data. However, in the primary analyses comparing RCT results within meta-analyses, there was no statistically significant association: unadjusted odds of greater than 50% mortality protection in high-JIF (> 5) journals were 1.4 (95% CI 0.42, 4.4) and adjusted, 2.5 (95% CI 0.6, 10). Elements of study quality were weakly, inconsistently, and not statistically significantly correlated with journal impact factor. Journal impact factor may have little to no association with study results, or methodological quality, but the evidence is very uncertain.Funding Information
- Queen's University (Summer Studentship)
This publication has 35 references indexed in Scilit:
- Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysisJournal of Clinical Epidemiology, 2013
- Methodological Reporting of Randomized Clinical Trials in Respiratory Research in 2010Respiratory Care, 2013
- Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort StudyPLoS Medicine, 2012
- Publication bias and time-trend bias in meta-analysis of bicycle helmet efficacy: A re-analysis of Attewell, Glase and McFadden, 2001Accident Analysis & Prevention, 2011
- RAPID VERSUS FULL SYSTEMATIC REVIEWS: VALIDITY IN CLINICAL PRACTICE?Anz Journal of Surgery, 2008
- Factors related to the frequency of citation of epidemiologic publicationsEpidemiologic Perspectives & Innovations, 2008
- Contradicted and Initially Stronger Effects in Highly Cited Clinical ResearchJAMA, 2005
- Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?The Lancet, 1998
- Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trialsJAMA, 1995
- Bias in analytic researchJournal of Chronic Diseases, 1979