Answers to 20 Questions About Interrater Reliability and Interrater Agreement
Top Cited Papers
- 28 November 2007
- journal article
- research article
- Published by SAGE Publications in Organizational Research Methods
- Vol. 11 (4), 815-852
- https://doi.org/10.1177/1094428106296642
Abstract
The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.Keywords
This publication has 47 references indexed in Scilit:
- Interrater Agreement Reconsidered: An Alternative to the rwg IndicesOrganizational Research Methods, 2005
- Estimating Interrater Agreement with the Average Deviation Index: A User’s GuideOrganizational Research Methods, 2002
- Statistical properties of the rWG(J) index of agreement.Psychological Methods, 2001
- Reducing the Effects of Performance Expectations on Behavioral RatingsOrganizational Behavior and Human Decision Processes, 2000
- On Average Deviation Indices for Estimating Interrater AgreementOrganizational Research Methods, 1999
- Group Size, ICC Values, and Group-Level Correlations: A SimulationOrganizational Research Methods, 1998
- Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models.Journal of Applied Psychology, 1998
- Leader-Member Exchange Model of Leadership: A Critique and Further DevelopmentAcademy of Management Review, 1986
- Ubiquitous halo.Psychological Bulletin, 1981
- On various intraclass correlation reliability coefficients.Psychological Bulletin, 1976