Look before you leap and don’t put all your eggs in one basket

Abstract

This paper’s aim is to draw attention to the pitfalls that novice and, sometimes, experienced researchers fall into when undertaking quantitative data analysis in the health and social sciences, and to offer some guidance as to how such pitfalls might be avoided. Many health and social science students are routinely instructed that the procedure for undertaking data analysis in quantitative research is as follows: specify hypotheses; collect data and enter it into a computerised statistical package; run various statistical procedures; examine the computer outputs for p-values that are statistically significant. If significant differences are found, jubilation often exists because statistically significant results are deemed to be a clear indicator that something worthwhile (and publishable) has been discovered. This paper argues that this approach has two major oversights: a failure to explore the raw data prior to analysis and an overdependence on p-values. Both of these oversights are routinely present in much health and social-science research, and both create problems for scientific rigour. Researchers need to exercise caution (‘look before you leap’) and prudence (‘don’t put all your eggs in one basket’) when undertaking quantitative data analyses. Caution demands that, prior to full data analysis, researchers employ procedures such as data cleaning, data screening and exploratory data analysis. Prudence demands that researchers see p-values for their true worth, which exists only within the context of statistical theory, confidence intervals, effect sizes and the absolute meaning of statistical significance.