Add-Remove-Confirm: Crowdsourcing synset cleansing

Thesaurus is a crucial resource for many natural language processing and artificial intelligence problems, which require common sense reasoning. It is becoming highly topical to put special effort to ensure the high quality of synsets when a thesaurus is created collaboratively by non-expert annotators. This paper proposes Add-Remove-Confirm, a novel workflow for crowdsourcing synset cleansing. The present workflow has been empirically evaluated using a Russian thesaurus created through crowdsourcing showing that it does improve the synset quality as according to the expert assessment with high level of agreement.