A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges
- 1 February 2004
- journal article
- other
- Published by SAGE Publications in Educational and Psychological Measurement
- Vol. 64 (1), 62-70
- https://doi.org/10.1177/0013164403260195
Abstract
This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson’s multivariate generalization of Cohen’s kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets.Keywords
This publication has 10 references indexed in Scilit:
- A Measure of Agreement for Interval or Nominal Multivariate ObservationsEducational and Psychological Measurement, 2001
- Bivariate Coefficients of Agreement among any Number of ObserversEducational and Psychological Measurement, 1993
- A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple RatersEducational and Psychological Measurement, 1988
- A review of reliability procedures for measuring observer agreementContemporary Educational Psychology, 1984
- A One-Way Components of Variance Model for Categorical DataBiometrics, 1977
- Kappa revisited.Psychological Bulletin, 1977
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Assessing the Accuracy of Multivariate ObservationsJournal of the American Statistical Association, 1966
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- ON THE CALCULATION OF INTRA-CLASS AND INTER-CLASS COEFFICIENTS OF CORRELATION FROM CLASS MOMENTS WHEN THE NUMBER OF POSSIBLE COMBINATIONS IS LARGEBiometrika, 1913