Sample Size for Cognitive Interview Pretesting

Abstract

Every cognitive interview pretest designer must decide how many interviews need to be conducted. With little theory or empirical research to guide the choice of sample size, practitioners generally rely on the examples of other studies and their own experience or preferences. We investigated pretest sample size both theoretically and empirically. Using a model of the relationship of sample size to question problem prevalence, detection power of the cognitive interview technique, and probability of observing a problem, we computed the sample size necessary, under varying conditions, to detect problems. Under a range of plausible values for the model parameters, we found that additional problems continued to be detected as sample size increased. We also report on an empirical study that simulated the number of problems detected at different sample sizes. Multiple outcome measures showed a strong positive relationship between sample size and problem detection; serious problems that were not detected in small samples were consistently observed in larger samples. We discuss the implications of these findings for practice and for additional research.