Crowdsourcing Experiment Designs for Chinese Word Sense Annotation

Crowdsourcing Experiment Designs for Chinese Word Sense Annotation ⿈黃資勻匀 Tzu-Yun Huang 國⽴立臺灣⼤大學語⾔言學研究所 Graduate Institute of Linguistics National Taiwan University r02142006@ntu.edu.tw 李佳臻 Chia-Chen, Lee 國⽴立臺灣⼤大學語⾔言學研究所 Graduate Institute of Linguistics National Taiwan University r03142008@ntu.edu.tw 李冠緯 Guan-Wei, Li 國⽴立臺灣⼤大學物理系碩⼠士班 Master’s Program of Institute of Physics National Taiwan University r04222065@ntu.edu.tw 吳⼩小涵 Hsiao-Han, Wu 國⽴立臺灣⼤大學語⾔言學研究所 Graduate Institute of Linguistics National Taiwan University r04142010@ntu.edu.tw 李韶曼 Shao-Man, Lee 國⽴立臺灣⼤大學法律學系博⼠士班 College of Law National Taiwan University d02a21006@ntu.edu.tw 謝舒凱 Shu-Kai, Hsieh 國⽴立臺灣⼤大學語⾔言所副教授 Graduate Institute of Linguistics National Taiwan University shukaihsieh@ntu.edu.tw The 2016 Conference on Computational Linguistics and Speech Processing ROCLING 2016, pp. 82-99  The Association for Computational Linguistics and Chinese Language Processing 82 Abstract This paper tries to demonstrate our exploratory efforts in tackling with the “high accuracy-low quantity” problem of human word sense annotation task in Chinese, and ultimately reach the goal of automatic word sense annotation. Our proposed annotation architecture consists of explicit and implicit aspects of of crowdsourcing approach. Explicit method focuses on the general issues of crowdsourcing and made adjustments on current MTurk framework. Implicit method concentrates on the idea of Game with a Purpose (GWAP) design, which originates from a well-known video game Super Mario. Keywords: WSD, Crowdsourcing, GWAP, Machine learning, Chinese Wordnet 1. Introduction Sense-aware system has become central to many NLP and related intelligent systems. The core technique involved is the Word Sense Disambiguation (WSD) which can determine the proper sense of each word in varied contexts. Current WSD models rely largely on gold standard data from manual annotation that has been suffering from the problems of high accuracy, low quantity and low efficiency. This paper aims to sketch a preliminary blueprint of (word) sense annotation service by resorting to crowdsourcing (CS) approaches tailored for the Chinese WSD task. Over the past years, crowdsourcing is an emerging collaborative way for collecting annotated corpus data and other types of language resources, with the advantages of being able to greatly increase the quantity and reduce time-cost by distribute the work to the public. Current implementations of crowdsourcing platforms include MTurks (e.g., Amazon Mechanical Turk; CrowdFlower), Game with a Purpose (GWAP) and Altruistic (or volunteer-based) crowdsourcing (e.g., Crowdcrafting). Although the explicit crowdsourcing method such as MTurks has been applied for years on several renowned platforms such as Yahoo!Answer, Quora, and so forth, several problems remain unsolved; for example, the recruitment of annotators, the annotator quality, and the design of the 83 platforms for the recruitment. Inspired by the CrowdTruth project , we propose an 1 internal-external adjusted framework to increase the ground-truth quality in the context of semantic annotation task. The explicit crowdsourcing has tackled with the main problems discovered in manual annotation; however, issue such expanses and interested- oriented bias still remain unsolved. Thus led to our second design, the implicit crowd- sourcing-game. GWAP design for annotations is not as common as the explicit approach since it is difficult to make an annotation game “interesting” and collect the required data in limited time. However, we assume that the implicit approach will become the …

Content truncated to 500 words