Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings
- 11 August 2009
- journal article
- research article
- Published by Taylor & Francis Ltd in Communication Monographs
- Vol. 76 (3), 286-302
- https://doi.org/10.1080/03637750903074685
Abstract
Meta-analysis involves cumulating effects across studies in order to qualitatively summarize existing literatures. A recent finding suggests that the effect sizes reported in meta-analyses may be negatively correlated with study sample sizes. This prediction was tested with a sample of 51 published meta-analyses summarizing the results of 3,602 individual studies. The correlation between effect size and sample size was negative in almost 80 percent of the meta-analyses examined, and the negative correlation was not limited to a particular type of research or substantive area. This result most likely stems from a bias against publishing findings that are not statistically significant. The primary implication is that meta-analyses may systematically overestimate population effect sizes. It is recommended that researchers routinely examine the n–r scatter plot and correlation, or some other indication of publication bias and report this information in meta-analyses.Keywords
This publication has 68 references indexed in Scilit:
- Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneityStatistics in Medicine, 2007
- Effects of music therapy for children and adolescents with psychopathology: a meta‐analysisJournal of Child Psychology and Psychiatry, 2004
- Adjusting for publication bias in the presence of heterogeneityStatistics in Medicine, 2003
- A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-AnalysisJournal of the American Statistical Association, 2000
- Reducing dating anxiety: A meta‐analysisCommunication Reports, 1998
- Comparing the persuasiveness of narrative and statistical evidence using meta‐analysisCommunication Research Reports, 1997
- How strong is the relationship between prejudice and discrimination? A meta-analytic answerInternational Journal of Intercultural Relations, 1996
- The role of educational briefings in mitigating effects of experimental exposure to violent sexually explicit material: A meta‐analysisThe Journal of Sex Research, 1996
- Meta‐analysis of self‐report data on the effectiveness of public speaking anxiety treatment techniquesCommunication Education, 1989
- The file drawer problem and tolerance for null results.Psychological Bulletin, 1979