DocumentCode :
2208705
Title :
Active Learning from Multiple Noisy Labelers with Varied Costs
Author :
Zheng, Yaling ; Scott, Stephen ; Deng, Kun
Author_Institution :
Dept. of Comput. Sci. & Eng., Univ. of Nebraska-Lincoln, Lincoln, NE, USA
fYear :
2010
fDate :
13-17 Dec. 2010
Firstpage :
639
Lastpage :
648
Abstract :
In active learning, where a learning algorithm has to purchase the labels of its training examples, it is often assumed that there is only one labeler available to label examples, and that this labeler is noise-free. In reality, it is possible that there are multiple labelers available (such as human labelers in the online annotation tool Amazon Mechanical Turk) and that each such labeler has a different cost and accuracy. We address the active learning problem with multiple labelers where each labeler has a different (known) cost and a different (unknown) accuracy. Our approach uses the idea of adjusted cost, which allows labelers with different costs and accuracies to be directly compared. This allows our algorithm to find low-cost combinations of labelers that result in high-accuracy labelings of instances. Our algorithm further reduces costs by pruning under-performing labelers from the set under consideration, and by halting the process of estimating the accuracy of the labelers as early as it can. We found that our algorithm often outperforms, and is always competitive with, other algorithms in the literature.
Keywords :
data mining; learning (artificial intelligence); active learning algorithm; high accuracy labeling; label purchase; low cost combination; multiple noisy labelers; noise-free labeler; under performing labeler pruning; active learning; adjusted cost; algorithms; multiple labelers; noisy labelers;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Data Mining (ICDM), 2010 IEEE 10th International Conference on
Conference_Location :
Sydney, NSW
ISSN :
1550-4786
Print_ISBN :
978-1-4244-9131-5
Electronic_ISBN :
1550-4786
Type :
conf
DOI :
10.1109/ICDM.2010.147
Filename :
5694018
Link To Document :
بازگشت