On Average Deviation Indices for Estimating Interrater Agreement

Abstract

In this investigation, the authors report the results of two studies designed to investigate the efficacy of two proposed indices of interrater agreement based on average deviations from the mean and from the median (ADM and ADMd, respectively). Using survey response data collected from 6,549 sales employees in 119 stores of a national retail company, Study 1 compared the results of six interrater agreement indices across four types of Likert-type response scales (i.e., 5-, 6-, 7-, and 11-point scales). The results indicated that the AD indices were highly correlated with an index of proportional agreement and with within-group interrater agreement indices. Study 2, based on survey data collected from 4,158 sales employees in 109 other stores of this company, constructively replicated Study 1 and examined the consistency of interrater agreement decisions across six indices with respect to a priori decision rules. Study 2 results also supported the use of AD indices. Practical issues concerning the use of AD indices for estimating interrater agreement and future research directions are discussed.