Combining active learning suggestions
Open Access
- 23 July 2018
- journal article
- research article
- Published by PeerJ in PeerJ Computer Science
- Vol. 4, e157
- https://doi.org/10.7717/peerj-cs.157
Abstract
We study the problem of combining active learning suggestions to identify informative training examples by empirically comparing methods on benchmark datasets. Many active learning heuristics for classification problems have been proposed to help us pick which instance to annotate next. But what is the optimal heuristic for a particular source of data? Motivated by the success of methods that combine predictors, we combine active learners with bandit algorithms and rank aggregation methods. We demonstrate that a combination of active learners outperforms passive learning in large benchmark datasets and removes the need to pick a particular active learner a priori. We discuss challenges to finding good rewards for bandit approaches and show that rank aggregation performs well.Funding Information
- Australian Commonwealth Government’s Cooperative Research Centers Programme
- Australian Research Council Centre of Excellence for All-sky Astrophysics (CAASTRO) (CE110001020)
This publication has 16 references indexed in Scilit:
- Kullback–Leibler upper confidence bounds for optimal sequential allocationThe Annals of Statistics, 2013
- Active LearningSynthesis Lectures on Artificial Intelligence and Machine Learning, 2012
- A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election methodSocial Choice and Welfare, 2010
- Exploration–exploitation tradeoff using variance estimates in multi-armed banditsTheoretical Computer Science, 2009
- SmcHD1, containing a structural-maintenance-of-chromosomes hinge domain, has a critical role in X inactivationNature Genetics, 2008
- Active learning for logistic regression: an evaluationMachine Learning, 2007
- The Nonstochastic Multiarmed Bandit ProblemSIAM Journal on Computing, 2002
- Active Hidden Markov Models for Information ExtractionLecture Notes in Computer Science, 2001
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948
- ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLESBiometrika, 1933