The Validity and Reliability of Survey Questions

Abstract

Inspired by the research of Frank Andrews on the reliability and validity of survey questions, a large-scale research project was conducted in the Netherlands. The project was comprised of two different stages. For this project, more than 600 survey questions were included in different surveys according to a multitrait-multimethod design. The resulting data were analyzed in two steps. In the first step, estimates of validity and reliability were obtained for each question. The second step was a meta-analysis of the variation in data quality found in the first step. This variation was related to question-specific characteristics, response scale characteristics, context characteristics, and design characteristics. The article describes how the results of this study can be of practical use. In addition, the authors compare them to results of similar studies in the United States, Austria, and other Western, Central, and Eastern European countries.