The effect of three basic task features on the sensitivity of acceptability judgment tasks

Abstract

Sprouse and Almeida (2017) provide a first systematic investigation of the sensitivity of four acceptability judgment tasks. In this project, we build on these results by decomposing those tasks into three constituent task features (single versus joint presentation, number of response options, and use of response labels), and explore the consequences of those task features on the sensitivity of acceptability judgment experiments. We present 6 additional experiments (for a total of 10) designed to probe the effect of those task features on sensitivity, both independently and in combination. Our results suggest three notable conclusions: (i) there is a clear advantage to joint presentation of theoretically-related sentence types, regardless of the type of response scale used in the experiment; (ii) tasks involving a continuous slider (which have an infinite number of response options, and few labels) offer good sensitivity, while relying solely on spatial reasoning rather than numeric reasoning; and (iii) there are a number of subtle interactions among the three task features that may warrant further investigation. We discuss the potential benefits and concerns of each of these features in detail, along with the relevance of these findings for deciding how to investigate both simple and higher-order acceptability contrasts.