The Use of Subsets of Test Questions in an Angoff Standard-Setting Method

Abstract

In an Angoff standard-setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item constituting the test. In many cases, these item performance estimates are made twice, with information shared with the judges between estimates. Especially for long tests, this estimation process can be time consuming and fatiguing for the participants. This study addressed the possibilities of reducing the number of items individual judges consider in an Angoff standard-setting study. The results of this study suggest that 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study. This could result in substantial savings of time and resources not only for the agencies that have been carrying out this activity but also for the practitioners (judges) who participate in standard-setting studies.