A critical investigation of recall and precision as measures of retrieval system performance
- 1 July 1989
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 7 (3), 205-229
- https://doi.org/10.1145/65943.65945
Abstract
Recall and precision are often used to evaluate the effectiveness of information retrieval systems. They are easy to define if there is a single query and if the retrieval result generated for the query is a linear ordering. However, when the retrieval results are weakly ordered, in the sense that several documents have an identical retrieval status value with respect to a query, some probabilistic notion of precision has to be introduced. Relevance probability, expected precision, and so forth, are some alternatives mentioned in the literature for this purpose. Furthermore, when many queries are to be evaluated and the retrieval results averaged over these queries, some method of interpolation of precision values at certain preselected recall levels is needed. The currently popular approaches for handling both a weak ordering and interpolation are found to be inconsistent, and the results obtained are not easy to interpret. Moreover, in cases where some alternatives are available, no comparative analysis that would facilitate the selection of a particular strategy has been provided. In this paper, we systematically investigate the various problems and issues associated with the use of recall and precision as measures of retrieval system performance. Our motivation is to provide a comparative analysis of methods available for defining precision in a probabilistic sense and to promote a better understanding of the various issues involved in retrieval performance evaluation.This publication has 17 references indexed in Scilit:
- Evaluation of information retrieval systems: A decision theory approachJournal of the American Society for Information Science, 1978
- A General Mathematical Model for Information Retrieval SystemsThe Library Quarterly, 1976
- On selecting a measure of retrieval effectiveness part II. Implementation of the philosophyJournal of the American Society for Information Science, 1973
- Distance between sets as an objective measure of retrieval effectivenessInformation Storage and Retrieval, 1973
- On selecting a measure of retrieval effectivenessJournal of the American Society for Information Science, 1973
- ON THE INVERSE RELATIONSHIP OF RECALL AND PRECISIONJournal of Documentation, 1972
- Evaluation problems in interactive information retrievalInformation Storage and Retrieval, 1970
- Evaluation Tests of Information Retrieval SystemsJournal of Documentation, 1970
- THE PARAMETRIC DESCRIPTION OF RETRIEVAL TESTSJournal of Documentation, 1969
- Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systemsAmerican Documentation, 1968