Interrater Agreement Reconsidered: An Alternative to the rwg Indices

Abstract

For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr ^wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a ^wg(1),is proposed. The authors derive thea ^wg(1)statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr ^wg(1)and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr ^wg(1)/rwg(J) when a uniform null is assumed,r ^wg(1)/rwg(J) indices that do not assume a uniform null,awg(1) / a ^wg(J)indices, and generalizability estimates of interrater agreement.