A Method for Analyzing Sparse Data Matrices in the Generalizability Theory Framework
Open Access
- 1 September 2002
- journal article
- research article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 26 (3), 321-338
- https://doi.org/10.1177/0146621602026003006
Abstract
In generalizability analyses, unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data. However, missing observations are common in operational performance assessment settings because of the nature of the assessment design. This article describes a procedure for overcoming the computational and technological limitations in analyzing data with missing observations by extracting data from a sparsely .lled data matrix into analyzable smaller subsets of data. This subdividing method is accomplished by creating data sets that exhibit structural designs that are common in generalizability analyses, namely, the crossed, MBIB, and nested designs. The validity of this subdividing method is examined using a Monte Carlo simulation. The method is demonstrated on an operational data set.Keywords
This publication has 19 references indexed in Scilit:
- Scoring Performance Assessments Based on JudgementsPublished by Springer Science and Business Media LLC ,2001
- Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 2000
- Monte Carlo Studies in Item Response TheoryApplied Psychological Measurement, 1996
- Generalizability Analyses of Work Keys Listening and Writing TestsEducational and Psychological Measurement, 1995
- The Vemont Portfolio Assessment Program: Findings and ImplicationsEducational Measurement: Issues and Practice, 1994
- Generalizability TheoryEducational Measurement: Issues and Practice, 1992
- Generalizability Theory: The Software ProblemJournal of Educational Statistics, 1985
- Balanced Incomplete Block Designs for Inter-Rater Reliability StudiesApplied Psychological Measurement, 1981
- THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†British Journal of Statistical Psychology, 1963
- Estimation of Variance and Covariance ComponentsBiometrics, 1953