Words, Numbers, and Visual Heuristics in Web Surveys: Is There a Hierarchy of Importance?

Abstract

Respondents follow simple heuristics in interpreting the visual features of questions. The authors carried out two experiments in two panels to investigate how the effect of visual heuristics affects the answers to survey questions. In the first experiment, the authors varied the distance between scale points in a 5-point scale to investigate whether respondents use the conceptual or visual midpoint of a scale. In the second experiment, the authors used different end point labels of a 5-point scale, by adding different shadings of color and numbers that differed both in sign and value (2 to -2), to study whether options that are similar of appearance are considered conceptually closer than when they are dissimilar in appearance. The authors predicted that there is a hierarchy of features that respondents attend to, with verbal labels taking precedence over numerical labels, and numerical labels taking precedence over visual cues. The results confirmed the hypothesis: the effect of spacing of response options and different end points was only apparent in polar point scales and not in fully labeled scales. In addition, this study on two panels, with one consisting of extremely trained respondents and the other of relatively fresh respondents, shows that trained respondents are affected by the distance between response options whereas relatively new respondents are not. To reduce the effect of visual cues, taking into account the robustness of results, the authors suggest it is better to use fully labeled 5-point scales in survey questions.