Contextual sensitivity in scientific reproducibility
Top Cited Papers
- 23 May 2016
- journal article
- Published by Proceedings of the National Academy of Sciences in Proceedings of the National Academy of Sciences of the United States of America
- Vol. 113 (23), 6454-6459
- https://doi.org/10.1073/pnas.1521897113
Abstract
In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher’s degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed “hidden moderators”) between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.Keywords
Funding Information
- National Science Foundation (1555131)
This publication has 80 references indexed in Scilit:
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth TellingPsychological Science, 2012
- Raise standards for preclinical cancer researchNature, 2012
- Statistical Conclusion Validity: Some Common Threats and Simple RemediesFrontiers in Psychology, 2012
- Failure to Replicate a Genetic Association May Provide Important Clues About Genetic ArchitecturePLOS ONE, 2009
- Experiencing Physical Warmth Promotes Interpersonal WarmthScience, 2008
- Why Current Publication Practices May Distort SciencePLoS Medicine, 2008
- Judging near and distant virtue and viceJournal of Experimental Social Psychology, 2008
- The Value Heuristic in Judgments of Relative FrequencyPsychological Science, 2008
- Why Most Published Research Findings Are FalsePLoS Medicine, 2005
- The file drawer problem and tolerance for null results.Psychological Bulletin, 1979