Challenges of Ground Truth Evaluation of Multi-target Tracking
- 1 June 2013
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops
- p. 735-742
- https://doi.org/10.1109/cvprw.2013.111
Abstract
Evaluating multi-target tracking based on ground truth data is a surprisingly challenging task. Erroneous or ambiguous ground truth annotations, numerous evaluation protocols, and the lack of standardized benchmarks make a direct quantitative comparison of different tracking approaches rather difficult. The goal of this paper is to raise awareness of common pitfalls related to objective ground truth evaluation. We investigate the influence of different annotations, evaluation software, and training procedures using several publicly available resources, and point out the limitations of current definitions of evaluation metrics. Finally, we argue that the development an extensive standardized benchmark for multi-target tracking is an essential step toward more objective comparison of tracking approaches.Keywords
This publication has 20 references indexed in Scilit:
- A multi-view annotation tool for people detection evaluationPublished by Association for Computing Machinery (ACM) ,2012
- Level-set person segmentation and tracking with multi-region appearance models and top-down shape informationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- Globally optimal solution to multi-object tracking with merged measurementsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- Globally-optimal greedy algorithms for tracking a variable number of objectsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- PETS2010 and PETS2009 Evaluation of Results Using Individual Ground Truthed Single ViewsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2010
- A mobile vision system for robust multi-person trackingPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2008
- Multicamera People Tracking with a Probabilistic Occupancy MapIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- LabelMe: A Database and Web-Based Tool for Image AnnotationInternational Journal of Computer Vision, 2007
- A Comparison and Evaluation of Multi-View Stereo Reconstruction AlgorithmsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2006
- Evaluating Multi-Object TrackingPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2006