Agreement and Information in the Reliability of Coding
Top Cited Papers
Open Access
- 1 April 2011
- journal article
- research article
- Published by Taylor & Francis Ltd in Communication Methods and Measures
- Vol. 5 (2), 93-112
- https://doi.org/10.1080/19312458.2011.568376
Abstract
Coefficients that assess the reliability of data-making processes, that is, coding text, transcribing interviews, or categorizing observations into analyzable terms, are mostly conceptualized in terms of the agreement a set of coders, observers, judges, or measuring instruments exhibit. When variation is low, reliability coefficients reveal their dependency on an often neglected phenomenon, the amount of information that reliability data provide about the reliability of the coding process or the data it generates. This paper explores the concept of reliability, simple agreement, three conceptions of chance to correct that agreement, and sources of information deficiency, and it develops two measures of information about reliability, akin to the power of a statistical test, intended as a companion to traditional reliability coefficients, especially Krippendorff's (2004 Krippendorff, K. 2004. Content analysis: An introduction to its methodology, 2nd, Thousand Oaks, CA: Sage. [Google Scholar] , pp. 221–250; Hayes & Krippendorff, 2007 Hayes, A. F. and Krippendorff, K. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1: 77–89. [Taylor & Francis Online] [Google Scholar] ) alpha.Keywords
This publication has 9 references indexed in Scilit:
- Systematic and Random Disagreement and the Reliability of Nominal DataCommunication Methods and Measures, 2008
- Answering the Call for a Standard Reliability Measure for Coding DataCommunication Methods and Measures, 2007
- Another look at interrater agreement.Psychological Bulletin, 1988
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- Reliability of Content Analysis: The Case of Nominal Scale CodingPublic Opinion Quarterly, 1955
- Measures of Association for Cross ClassificationsJournal of the American Statistical Association, 1954
- Communications Through Limited Response QuestioningPublic Opinion Quarterly, 1954
- Coefficient alpha and the internal structure of testsPsychometrika, 1951
- VIII. Mathematical contributions to the theory evolution. —IX. On principle of homotyposis and its relation, the variability of the individual, and to that of the race. Part I. —Homotyposis in the vegetable KingdomPhilosophical Transactions of the Royal Society A, 1901