Statistical analysis of the National Institutes of Health peer review system
- 12 August 2008
- journal article
- Published by Proceedings of the National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 105 (32), 11076-11080
- https://doi.org/10.1073/pnas.0804538105
Abstract
A statistical model is proposed for the analysis of peer-review ratings of R01 grant applications submitted to the National Institutes of Health. Innovations of this model include parameters that reflect differences in reviewer scoring patterns, a mechanism to account for the transfer of information from an application's preliminary ratings and group discussion to final ratings provided by all panel members and posterior estimates of the uncertainty associated with proposal ratings. Application of this model to recent R01 rating data suggests that statistical adjustments to panel rating data would lead to a 25% change in the pool of funded proposals. Viewed more broadly, the methodology proposed in this article provides a general framework for the analysis of data collected interactively from expert panels through the use of the Delphi method and related procedures.Keywords
This publication has 17 references indexed in Scilit:
- Covariates of the Rating Process in Hierarchical Models for Multiple Ratings of Test ItemsJournal of Educational and Behavioral Statistics, 2007
- Performance Indicators: Good, Bad, and UglyJournal of the Royal Statistical Society Series A: Statistics in Society, 2004
- The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment DataJournal of Educational and Behavioral Statistics, 2002
- Latent variables in psychology and the social sciencesAnnual Review of Psychology, 2001
- The Rater Bundle ModelJournal of Educational and Behavioral Statistics, 2001
- Univariate and multirater ordinal cumulative link regression with covariate specific cutpointsThe Canadian Journal of Statistics / La Revue Canadienne de Statistique, 2000
- On Bayesian Analysis of Multirater Ordinal Data: An Application to Automated Essay GradingJournal of the American Statistical Association, 1996
- On Bayesian Analysis of Multirater Ordinal Data: An Application to Automated Essay GradingJournal of the American Statistical Association, 1996
- Bayesian Analysis of Binary and Polychotomous Response DataJournal of the American Statistical Association, 1993
- Bayesian Analysis of Binary and Polychotomous Response DataJournal of the American Statistical Association, 1993