Commingled Samples: A Neglected Source of Bias in Reliability Analysis

Abstract

Reliability is a property of test scores from individuals who have been sampled from a well-defined population. Reliability indices, such as coefficient α and related formulas for internal consistency reliability (KR-20, Hoyt's reliability), yield lower bound reliability estimates when (a) subjects have been sampled from a single population and when (b) test items are congeneric (i.e., when items are sampled from a single latent dimension). However, when samples are commingled—that is, when they are composed of scores that are drawn from multiple populations— coefficient α and related indices can be severely biased. In most cases the bias inflates α; in other cases α is attenuated. Equations are derived for elucidating this bias in two-group mixture distributions.