Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection
- 1 June 2015
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
We introduce tools and methodologies to collect high quality, large scale fine-grained computer vision datasets using citizen scientists - crowd annotators who are passionate and knowledgeable about specific domains such as birds or airplanes. We worked with citizen scientists and domain experts to collect NABirds, a new high quality dataset containing 48,562 images of North American birds with 555 categories, part annotations and bounding boxes. We find that citizen scientists are significantly more accurate than Mechanical Turkers at zero cost. We worked with bird experts to measure the quality of popular datasets like CUB-200-2011 and ImageNet and found class label error rates of at least 4%. Nevertheless, we found that learning algorithms are surprisingly robust to annotation errors and this level of training data corruption can lead to an acceptably small increase in test error if the training set has sufficient size. At the same time, we found that an expert-curated high quality test set like NABirds is necessary to accurately measure the performance of fine-grained computer vision systems. We used NABirds to train a publicly available bird recognition service deployed on the web site of the Cornell Lab of Ornithology.Keywords
This publication has 23 references indexed in Scilit:
- DeepFace: Closing the Gap to Human-Level Performance in Face VerificationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2014
- Repeated labeling using multiple noisy labelersData Mining and Knowledge Discovery, 2013
- Online crowdsourcing: Rating annotators and obtaining cost-effective labelsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2010
- Motivations of Wikipedia content contributorsComputers in Human Behavior, 2010
- reCAPTCHA: Human-Based Character Recognition via Web Security MeasuresScience, 2008
- Get another label? improving data quality and data mining using multiple, noisy labelersPublished by Association for Computing Machinery (ACM) ,2008
- Utility data annotation with Amazon Mechanical TurkPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2008
- Games with a PurposeComputer, 2006
- Motivations of contributors to WikipediaACM SIGCAS Computers and Society, 2006
- Pay Enough or Don't Pay at All*The Quarterly Journal of Economics, 2000