Quality Control for Crowdsourced Bilingual Dictionary in Low-Resource Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6590–6596 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6590 Quality Control for Crowdsourced Bilingual Dictionary in Low-Resource Languages Hiroki Chida, Yohei Murakami, Mondheera Pituxcoosuvarn Ritsumeikan University 1–1–1 Noji-Higashi, Kusatsu, Shiga, 525–8577 Japan is0363xp@ed.ritsumei.ac.jp, yohei@fc.ritsumei.ac.jp, mondheera@fc.ritsumei.ac.jp Abstract In conventional bilingual dictionary creation by using crowdsourcing, the main method is to ask multiple workers to translate the same words or sentences and take a majority vote. However, when this method is applied to the creation of bilingual dictionaries for low-resource languages with few speakers, many low-quality workers are expected to participate in the majority voting, which makes it difficult to maintain the quality of the evaluation by the majority voting. Therefore, we apply an effective aggregation method using a hyper question, which is a set of single questions, for quality control. Furthermore, to select high- quality workers, we design a task-allocation method based on the reliability of workers which is evaluated by their work results. Keywords: crowdsourcing，bilingual dictionary，quality control，task assignment 1. Introduction Recently, crowdsourcing is becoming mainstream to create language resources including bilingual dictio- naries. Crowdsourcing is a scheme for requesting work from a large and open group of people via the in- ternet, and it can be used to order a large number of works that require human labor. Crowdsourcing is especially used to request relatively difficult tasks for computers, but not so difficult for humans. How- ever, in crowdsourcing, where the tasks are executed by an unspecified number of workers the abilities of whom vary, it is difficult to guarantee the quality of the execution results. Especially in the case of bilin- gual dictionaries creation between low-resource lan- guages (Murakami, 2019), the number of people who can speak multiple low-resource languages is limited, and the average ability of workers is low. This results in the method of assigning the same task to multiple workers and using majority voting has a high possi- bility of obtaining wrong answers, and quality control cannot be performed well. Therefore, we aim to improve quality in an environ- ment with a small number of highly reliable workers by using an answer aggregation method on hyper ques- tions (multiple tasks considered together as one task). Since workers with high ability tend to agree on the answers to hyper questions, the method increases the possibility that workers with high ability will be in the majority. To this end, we address the following two problems. Selecting highly reliable evaluators In the answer aggregation method on hyper ques- tions, it is assumed that a small number of high- quality workers are involved. Therefore, it is nec- essary to select highly reliable evaluators from a crowd. Reducing the number of tasks Even if a worker is able to correctly evaluate whether a bilingual text is correct or not, in the case of wrong bilingual texts, the worker may have to redo the translation, which increases the number of tasks. For these …

Content truncated to 500 words