The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability : A Monte Carlo Investigation
- 1 March 1985
- journal article
- research article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 9 (1), 31-36
- https://doi.org/10.1177/014662168500900103
Abstract
A computer simulation study was designed to in vestigate the extent to which the interrater reliability of a clinical scale is affected by the number of cate gories or scale points (2, 3, 4, ... ,100). Results in dicate that reliability increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale poirits is increased to as many as 100. These findings hold under the follow ing conditions: (1) The research investigator has insuf ficient a priori knowledge to use as a reliable guide line for deciding on an appropriate number of scale points to employ, and (2) the dichotomous and ordinal categories being considered all have an underlying metric or continuous scale format.Keywords
This publication has 31 references indexed in Scilit:
- The Cost of DichotomizationApplied Psychological Measurement, 1983
- A Monte Carlo study of factors affecting three indices of composite scale reliability.Journal of Applied Psychology, 1977
- Effect of the number of scale points on reliability: A Monte Carlo approach.Journal of Applied Psychology, 1975
- The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of ReliabilityEducational and Psychological Measurement, 1973
- Large sample standard errors of kappa and weighted kappa.Psychological Bulletin, 1969
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- Coefficient alpha and the internal structure of testsPsychometrika, 1951
- Optimal refinement of the rating scale.Journal of Applied Psychology, 1939
- On the Loss of Reliability in Ratings Due to Coarseness of the Scale.Journal of Experimental Psychology, 1924