The knowledge system underpinning healthcare is not fit for purpose and must change

Abstract

The medical literature is biased and inundated with poor quality trials. Ian Roberts and colleagues explain how these problems affect systematic reviews and how they might be overcome Information on the effectiveness and safety of healthcare should be valid, precise, up to date, clear, and freely available. Currently none of these criteria are fully satisfied, and Cochrane systematic reviews are not the solution. In this article we explain why the knowledge system for healthcare is unfit for purpose and suggest how it should change. Because the medical literature contains a biased sample of trials, systematic reviews that are based on it are also biased.1 Despite decades of exhortation about trial publication, about half of all trials are unpublished, and even the most diligent efforts to synthesise the results from all (or an unbiased sample of) relevant trials are in vain.1 2 Even when trials are identified selective outcome reporting limits their validity.3 4 The literature is awash with low quality, underpowered, single centre trials and the trend is upwards. As Altman recognised over 20 years ago “much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them.”5 The systematic review movement is renowned for its obsessive zeal to find every published trial, irrespective of size or quality. These efforts have increased the accessibility of many trials that should never have been conducted. Their citation in reviews perpetuates the problem by contributing to the impact factor of the journals that published them. In the United Kingdom, the funding provided to Cochrane review groups is proportional to the number of trials included in reviews, creating a financial incentive to find and include every trial regardless of its quality. Most journal editors and systematic …