Convergent validation of peer review decisions using the h indexExtent of and reasons for type I and type II errors
- 1 July 2007
- journal article
- research article
- Published by Elsevier BV in Journal of Informetrics
- Vol. 1 (3), 204-213
- https://doi.org/10.1016/j.joi.2007.01.002
Abstract
Hirsch [Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572] has proposed the h index as a single-number criterion to evaluate the scientific output of a researcher. We investigated the convergent validity of decisions for awarding long-term fellowships to post-doctoral researchers as practiced by the Boehringer Ingelheim Fonds (B.I.F.) by using the h index. Our study examined 414 B.I.F. applicants (64 approved and 350 rejected) with a total of 1586 papers. The results of our study show that the applicants’ h indices correlate substantially with standard bibliometric indicators. Even though the h indices of approved B.I.F. applicants on average (arithmetic mean and median) are higher than those of rejected applicants (and with this, fundamentally confirm the validity of the funding decisions), the distributions of the h indices show in part overlaps that we categorized as type I error (falsely drawn approval) or type II error (falsely drawn rejection). Approximately, one-third of the decisions to award a fellowship to an applicant show a type I error, and about one-third of the decisions not to award a fellowship to an applicant show a type II error. Our analyses of possible reasons for these errors show that the applicant's field of study but not personal ties between the B.I.F. applicant and the B.I.F. can increase or decrease the risks for type I and type II errors.Keywords
This publication has 7 references indexed in Scilit:
- Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicantsScientometrics, 2006
- Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groupsScientometrics, 2006
- Using the h‐index to rank influential information scientistssJournal of the American Society for Information Science and Technology, 2006
- The h index and career assessment by numbersTrends in Ecology & Evolution, 2006
- An index to quantify an individual's scientific research outputProceedings of the National Academy of Sciences of the United States of America, 2005
- Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisionsScientometrics, 2005
- Measuring SciencePublished by Springer Science and Business Media LLC ,2004