Using Crowd Agreement for Wordnet Localization

Using Crowd Agreement for Wordnet Localization Amarsanaa Ganbold∗, Altangerel Chagnaa∗, G´abor Bella† ∗Department of Information and Computer Science, SEAS, National University of Mongolia Ikh surguuliin gudamj – 3, Sukhbaatar district, 14201 Ulaanbaatar, Mongolia †Department of Information and Computer Science, University of Trento via Sommarive, 5, 38123 Trento, Italy {amarsanaag, altangerel}@num.edu.mn, gabor.bella@unitn.it Abstract Building a wordnet from scratch is a huge task, especially for languages less equipped with pre-existing lexical resources such as thesauri or bilingual dictionaries. We address the issue of costliness of human supervision through crowdsourcing that offers a good trade-off between quality of output and speed of progress. In this paper, we demonstrate a two-phase crowdsourcing workflow that consists of a synset localization step followed by a validation step. Validation is performed using the inter-rater agreement metrics Fleiss’ kappa and Krippendorf’s alpha, which allow us to estimate the precision of the result, as well as to set a balance between precision and recall. In our experiment, 947 synsets were localized from English to Mongolian and evaluated through crowdsourcing with the precision of 0.74. Keywords: crowdsourcing evaluation, inter-rater agreement, synset localization, wordnet 1. Introduction Lexical-semantic resources like WordNet are a fundamen- tal resource for many NLP and semantic applications. Building such resources traditionally involves the collab- oration of a large number of professionals, such as psy- cholinguists and lexicographers, in order to obtain a high- quality end result. To mitigate the cost of WordNet con- struction, automatic and semi-automatic approaches have been successfully used, based on existing bilingual re- sources such as dictionaries and thesauri. Such methods, however, are hardly applicable to less-resourced languages that do not already have rich thesauri and dictionaries. In this paper we introduce an alternative approach of local- izing wordnet synsets through crowdsourcing. The process consists of two phases: in phase one, workers build synsets by contributing synonymous words while in phase two they validate the correctness of words provided earlier. Valida- tion results are combined and assessed using two alterna- tive statistical metrics of inter-rater agreement, the idea be- ing that higher levels of agreement corresponds to higher precision. We evaluate this hypothesis with the help of a gold standard corpus. The evaluation provides us with an insight on the efficiency of crowdsourcing and allows us to optimize the coefficient parameters of these metrics for increased precision. 2. Related Work Ontology localization, as described by Espinoza et al. (2009) and Cimiano et al. (2010), presents an alternative approach to addressing the cost of building lexical-semantic resources. An approach based on ontology label transla- tion (Arcan and Buitelaar, 2013) was developed to provide a knowledge-based extension to a statistical machine trans- lation system. However, automatically translated multilin- gual terms often suffer from quality issues. A preliminary crowdsourcing model for less-resourced languages (Ben- jamin and Radetzky, 2014) allows the building of lexicons with the help of Internet users. They concluded that ex- tensive manipulation and review by language experts was necessary in order to obtain high-quality linguistic data and to capture a large diversity of …

Content truncated to 500 words