Noise correction of image labeling in crowdsourcing

We investigate the methods of improving data quality, in terms of label accuracy, in the context of image labeling in crowdsourcing. First, we look at three consensus methods for inferring a ground-truth label from the multiple noisy labels obtained from crowdsourcing, i.e., Majority Voting (MV), Dawid Skene (DS), and KOS. We then apply three noise correction methods to correct labels inferred by these consensus methods, i.e., Polishing Labels (PL), Self-Training Correction (STC), and Cluster Correction (CC). Our experimental results show that the noise correction methods improve the labeling quality significantly.