Exploring the reliability of social and environmental disclosures content analysis

Abstract

This paper reports the results of an exploratory study of inter‐coder reliability of annual report social and environmental disclosures content analysis. Using the sentence‐based coding instruments and decision rules from Hackston and Milne (1996), this study reports the co‐agreement levels reached by three coders over five rounds of testing 49 annual reports. The study also provides a commentary on the implications of formal reliability analysis for past and future social and environmental disclosures content analyses, and the complexities of formal reliability measurement. The overall findings suggest that the coded output from inexperienced coders using the Hackston and Milne approach with little or no prior training can be relied on for aggregate total disclosures analysis. For more detailed sub‐category analysis, however, the findings suggest a period of training for the less experienced coders with at least 20 reports appears necessary before their coded output could be relied on.