The failing measurement of attitudes: How semantic determinants of individual survey responses come to replace measures of attitude strength
- 11 January 2018
- journal article
- research article
- Published by Springer Science and Business Media LLC in Behavior Research Methods
- Vol. 50 (6), 2345-2365
- https://doi.org/10.3758/s13428-017-0999-y
Abstract
The traditional understanding of data from Likert scales is that the quantifications involved result from measures of attitude strength. Applying a recently proposed semantic theory of survey response, we claim that survey responses tap two different sources: a mixture of attitudes plus the semantic structure of the survey. Exploring the degree to which individual responses are influenced by semantics, we hypothesized that in many cases, information about attitude strength is actually filtered out as noise in the commonly used correlation matrix. We developed a procedure to separate the semantic influence from attitude strength in individual response patterns, and compared these results to, respectively, the observed sample correlation matrices and the semantic similarity structures arising from text analysis algorithms. This was done with four datasets, comprising a total of 7,787 subjects and 27,461,502 observed item pair responses. As we argued, attitude strength seemed to account for much information about the individual respondents. However, this information did not seem to carry over into the observed sample correlation matrices, which instead converged around the semantic structures offered by the survey items. This is potentially disturbing for the traditional understanding of what survey data represent. We argue that this approach contributes to a better understanding of the cognitive processes involved in survey responses. In turn, this could help us make better use of the data that such methods provide.This publication has 74 references indexed in Scilit:
- Private traits and attributes are predictable from digital records of human behaviorProceedings of the National Academy of Sciences of the United States of America, 2013
- Not Lost in Translation: Neural Responses Shared Across LanguagesJournal of Neuroscience, 2012
- The General Factor of Personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity studyJournal of Research in Personality, 2010
- The Perils of PartiallingAssessment, 2006
- 9. A Mathematical Approach to Categorization and Labeling of Qualitative Data: The Latent Categorization MethodSociological Methodology, 2004
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990
- Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information TechnologyMIS Quarterly, 1989
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorJournal of Marketing Research, 1981
- Construct validity in psychological tests.Psychological Bulletin, 1955
- Attitudes vs. ActionsSocial Forces, 1934