Analysis of rating scales: A pervasive problem in bilingualism research and a solution with Bayesian ordinal models
- 1 September 2021
- journal article
- review article
- Published by Cambridge University Press (CUP) in Bilingualism: Language and Cognition
- Vol. 24 (5), 842-848
- https://doi.org/10.1017/s1366728921000316
Abstract
Research in bilingualism often involves quantifying constructs of interest by the use of rating scales: for example, to measure language proficiency, dominance, or sentence acceptability. However, ratings are a type of ordinal data, which violates the assumptions of the statistical methods that are commonly used to analyse them. As a result, the validity of ratings is compromised and the ensuing statistical inferences can be seriously distorted. In this article, we describe the problem in detail and demonstrate its pervasiveness in bilingualism research. We then provide examples of how bilingualism researchers can employ an appropriate solution using Bayesian ordinal models. These models respect the inherent discreteness of ratings, easily accommodate non-normality, and allow modelling unequal psychological distances between response categories. As a result, they can provide more valid, accurate, and informative inferences about graded constructs such as language proficiency. Data and code are publicly available in an OSF repository at https://osf.io/grs8x.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (Project ID 317633480 – SFB 1287, Project Q.)
This publication has 29 references indexed in Scilit:
- Do People Have Insight Into Their Abilities? A MetasynthesisPerspectives on Psychological Science, 2014
- The construct of language proficiency in the study of bilingualism from a cognitive perspectiveBilingualism: Language and Cognition, 2012
- The syntax–discourse interface in near-native L2 acquisition: Off-line and on-line performanceBilingualism: Language and Cognition, 2009
- The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing Language Profiles in Bilinguals and MultilingualsJournal of Speech, Language, and Hearing Research, 2007
- Critical EvidencePsychological Science, 2003
- Age Constraints on Second-Language AcquisitionJournal of Memory and Language, 1999
- Self-reports: How the questions shape the answers.American Psychologist, 1999
- The Subject-Item Interaction in Itemmetric ResearchEducational and Psychological Measurement, 1977
- Science and StatisticsJournal of the American Statistical Association, 1976
- A statistical model for the analysis of ordinal level dependent variablesThe Journal of Mathematical Sociology, 1975