Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources
Top Cited Papers
Open Access
- 15 October 2019
- journal article
- research article
- Published by Wiley in Research Synthesis Methods
- Vol. 11 (2), 181-217
- https://doi.org/10.1002/jrsm.1378
Abstract
Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses), because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analysed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognise the requirements of evidence synthesis, and for academic journals to re‐assess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.Keywords
This publication has 89 references indexed in Scilit:
- Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enoughBMC Medical Research Methodology, 2013
- Is the coverage of google scholar enough to be used alone for systematic reviewsBMC Medical Informatics and Decision Making, 2013
- Reducing and meta-analysing estimates from distributed lag non-linear modelsBMC Medical Research Methodology, 2013
- Automated dose dispensing service for primary healthcare patients: a systematic reviewSystematic Reviews, 2013
- Impact of unlinked deaths and coding changes on mortality trends in the Swiss National CohortBMC Medical Informatics and Decision Making, 2013
- Decision aids for people facing health treatment or screening decisionsPublished by Wiley ,2011
- Why not just Google it? An assessment of information literacy skills in a biomedical science curriculumBMC Medical Education, 2011
- Veterinary students' usage and perception of video teaching resourcesBMC Medical Education, 2011
- The rate of growth in scientific publication and the decline in coverage provided by Science Citation IndexScientometrics, 2010