Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified

Abstract

It is generally accepted that meta-analyses should assess heterogeneity, which may be defined as the presence of variation in true effect sizes underlying the different studies. This assessment might be achieved by performing a statistical test for heterogeneity, by quantifying its magnitude, by quantifying its impact or by a combination of these. Patsopoulos, Evangelou and Ioannidis propose methods for examining the effect of excluding studies (or groups of studies) on an assessment of heterogeneity.¹ Their methods offer benefits over the sometimes practiced ‘leave one out’ approach to sensitivity analysis, by recognizing that the overall effect (against which heterogeneity is measured) changes each time an influential study is excluded. The authors offer a sequential approach (in which the overall effect and heterogeneity measure are re-estimated after the most influential study is removed at each iteration), and a combinatorial approach (in which groups of studies are removed). I suspect the sequential approach may often be performed informally in practice, whereby an obvious outlier is excluded, but another study then appears to be an outlier compared with the remaining studies and is in turn excluded. Of course, if heterogeneity permeates the entire set of studies, one might be tempted continue excluding studies to reduce heterogeneity until a single study remains. A predefined stopping rule (a ‘desired heterogeneity threshold’, in the authors’ terminology) may therefore appear to offer a useful way forward.