Apples to Oranges or Gala versus Golden Delicious?

Abstract

Nonprobability samples have gained mass popularity and use in many research circles, including market research and some political research. One justification for the use of nonprobability samples is that low response rate probability surveys have nothing significant to offer over and above a “well built” nonprobability sample. Utilizing an elemental approach, we compare a range of samples, weighting, and modeling procedures in an analysis that evaluates the estimated bias of various cross-tabulations of core demographics. Specifically, we compare a battery of bias related metrics for nonprobability panels, dual-frame telephone samples, and a high-quality in-person sample. Results indicate that there is roughly a linear trend, with nonprobability samples attaining the greatest estimated bias, and the in-person sample, the lowest. Results also indicate that the bias estimates vary widely for the nonprobability samples compared to either the telephone or in-person samples, which themselves tend to have consistently smaller amounts of estimated bias. Specifically, both weighted and unweighted dual-frame telephone samples were found to have about half the estimated bias compared to analogous nonprobability samples. Advanced techniques such as propensity weighting and sample matching did not improve these measures, and in some cases made matters worse. Implications for “fit for purpose” in survey research are discussed given these findings.