Title :
Utility data annotation with Amazon Mechanical Turk
Author :
Sorokin, Alexander ; Forsyth, David
Author_Institution :
Univ. of Illinois at Urbana-Champaign, Urbana, IL
Abstract :
We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.
Keywords :
Web services; Web sites; data handling; image processing; Amazon Mechanical Turk; annotation problems; utility data annotation; Centralized control; Computer vision; Costs; Feedback; Gold; HTML; Humans; Protocols; Quality assurance; Throughput;
Conference_Titel :
Computer Vision and Pattern Recognition Workshops, 2008. CVPRW '08. IEEE Computer Society Conference on
Conference_Location :
Anchorage, AK
Print_ISBN :
978-1-4244-2339-2
Electronic_ISBN :
2160-7508
DOI :
10.1109/CVPRW.2008.4562953
