What They See Is What We Get

Abstract

Several alternative response formats are available to the web survey designer, but the choice of format is often made with little consideration of measurement error. The authors experimentally explore three common response formats used in web surveys: a series of radio buttons, a drop box with none of the options initially displayed until the respondent clicks on the box, and a scrollable drop box with some of the options initially visible, requiring the respondent to scroll to see the remainder of the options. The authors reversed the order of the response options for half the sample. The authors find evidence of response order effects but stronger evidence that visible response options are endorsed more frequently, suggesting that visibility may be a more powerful effect than primacy in web surveys. The results suggest that the response format used in web surveys does affect the choices made by respondents.