A similarity measure for indefinite rankings
Top Cited Papers
- 23 November 2010
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 28 (4), 1-38
- https://doi.org/10.1145/1852102.1852106
Abstract
Ranked lists are encountered in research and daily life and it is often of interest to compare these lists even when they are incomplete or have only some members in common. An example is document rankings returned for the same query by different search engines. A measure of the similarity between incomplete rankings should handle nonconjointness, weight high ranks more heavily than low, and be monotonic with increasing depth of evaluation; but no measure satisfying all these criteria currently exists. In this article, we propose a new measure having these qualities, namely rank-biased overlap (RBO). The RBO measure is based on a simple probabilistic user model. It provides monotonicity by calculating, at a given depth of evaluation, a base score that is non-decreasing with additional evaluation, and a maximum score that is nonincreasing. An extrapolated score can be calculated between these bounds if a point estimate is required. RBO has a parameter which determines the strength of the weighting to top ranks. We extend RBO to handle tied ranks and rankings of different lengths. Finally, we give examples of the use of the measure in comparing the results produced by public search engines and in assessing retrieval systems in the laboratory.Keywords
This publication has 16 references indexed in Scilit:
- On rank correlation and the distance between rankingsPublished by Association for Computing Machinery (ACM) ,2009
- On rank correlation in information retrieval evaluationACM SIGIR Forum, 2007
- Methods for comparing rankings of search engine resultsComputer Networks, 2006
- Comparing rankings of search results on the WebInformation Processing & Management, 2005
- Space-Limited Ranked Query Evaluation Using Adaptive PruningLecture Notes in Computer Science, 2005
- Topic prediction based on comparative retrieval rankingsPublished by Association for Computing Machinery (ACM) ,2004
- Comparing Top k ListsSIAM Journal on Discrete Mathematics, 2003
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- Theory & Methods: Rank Correlation — an Alternative MeasureAustralian & New Zealand Journal of Statistics, 2000
- A Measure of Top-Down CorrelationTechnometrics, 1987