A randomized trial in a massive online open course shows people don’t know what a statistically significant relationship looks like, but they can learn
Open Access
- 16 October 2014
- Vol. 2, e589
- https://doi.org/10.7717/peerj.589
Abstract
Scatterplots are the most common way for statisticians, scientists, and the public to visually detect relationships between measured variables. At the same time, and despite widely publicized controversy, P-values remain the most commonly used measure to statistically justify relationships identified between variables. Here we measure the ability to detect statistically significant relationships from scatterplots in a randomized trial of 2,039 students in a statistics massive open online course (MOOC). Each subject was shown a random set of scatterplots and asked to visually determine if the underlying relationships were statistically significant at the P < 0.05 level. Subjects correctly classified only 47.4% (95% CI [45.1%–49.7%]) of statistically significant relationships, and 74.6% (95% CI [72.5%–76.6%]) of non-significant relationships. Adding visual aids such as a best fit line or scatterplot smooth increased the probability a relationship was called significant, regardless of whether the relationship was actually significant. Classification of statistically significant relationships improved on repeat attempts of the survey, although classification of non-significant relationships did not. Our results suggest: (1) that evidence-based data analysis can be used to identify weaknesses in theoretical procedures in the hands of average users, (2) data analysts can be trained to improve detection of statistically significant results with practice, but (3) data analysts have incorrect intuition about what statistically significant relationships look like, particularly for small effects. We have built a web tool for people to compare scatterplots with their corresponding p-values which is available here: http://glimmer.rstudio.com/afisher/EDA/.This publication has 13 references indexed in Scilit:
- Scientific method: Statistical errorsNature, 2014
- Self-Driven Mastery in Massive Open Online CoursesMOOCs FORUM, 2013
- The Perception of Correlation in ScatterplotsComputer Graphics Forum, 2010
- Statistical Inference After Model SelectionJournal of Quantitative Criminology, 2009
- Why Most Published Research Findings Are False: Problems in the AnalysisPLoS Medicine, 2007
- Researchers Misunderstand Confidence Intervals and Standard Error Bars.Psychological Methods, 2005
- A randomized comparison of patients’ understanding of number needed to treat and other common risk reduction formatsJournal of General Internal Medicine, 2003
- The Role of Numeracy in Understanding the Benefit of Screening MammographyAnnals of Internal Medicine, 1997
- Graphical Perception and Graphical Methods for Analyzing Scientific DataScience, 1985
- Variables on Scatterplots Look More Highly Correlated When the Scales Are IncreasedScience, 1982