Detecting and Correcting the Lies That Data Tell

Abstract

Because of the way in which data are typically analyzed and interpreted, they frequently lie to researchers, leading to conclusions that are not only false but more complex than the underlying reality. The several examples of this presented in this article illustrate the possibility that although data may appear to indicate complex phenomena at the surface structure level, the phenomena may be quite simple at the deep structure level, suggesting the possibility of applying Occam’s razor to achieve the scientific goal of parsimony. The approaches to data analysis described in this article may also lead to a solution to the serious problem of construct proliferation in psychology by demonstrating that many constructs are redundant with other existing constructs. The major obstacles to these outcomes are researchers' continued reliance on the use of statistical significance testing in data analysis and interpretation and the failure to correct for the distorting effects of sampling error, measurement error, and other artifacts. Some of these problems have been addressed by the now widespread use of meta-analysis, but examination of the meta-analyses appearing in Psychological Bulletin from 1978 to 2006 shows that most employ a statistically inappropriate model for meta-analysis (the fixed effects model) and that 90% do not correct for the biasing effects of measurement error. Hence, there is still a long way to go in the improvement of data analysis and interpretation methods.