Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2801 - 2813 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations Xin Zhang1, Guangwei Xu, Yueheng Sun2, Meishan Zhang3∗, Xiaobin Wang, Min Zhang3 1School of New Media and Communication, Tianjin University, China 2College of Intelligence and Computing, Tianjin University, China 3Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China {hsinz,yhs}@tju.edu.cn, {ahxgwOnePiece, mason.zms}@gmail.com czwangxiaobin@foxmail.com, zhangmin2021@hit.edu.cn Abstract Recent works of opinion expression identifi- cation (OEI) rely heavily on the quality and scale of the manually-constructed training cor- pus, which could be extremely difficult to sat- isfy. Crowdsourcing is one practical solution for this problem, aiming to create a large-scale but quality-unguaranteed corpus. In this work, we investigate Chinese OEI with extremely- noisy crowdsourcing annotations, constructing a dataset at a very low cost. Following Zhang et al. (2021), we train the annotator-adapter model by regarding all annotations as gold- standard in terms of crowd annotators, and test the model by using a synthetic expert, which is a mixture of all annotators. As this annotator- mixture for testing is never modeled explicitly in the training phase, we propose to generate synthetic training samples by a pertinent mixup strategy to make the training and testing highly consistent. The simulation experiments on our constructed dataset show that crowdsourcing is highly promising for OEI, and our proposed annotator-mixup can further enhance the crowd- sourcing modeling. 1 Introduction Opinion mining is a fundamental topic in the natu- ral language processing (NLP) community, which has received great attention for decades (Liu and Zhang, 2012). Opinion expression identification (OEI) is a standard task of opinion mining, which aims to recognize the text spans that express partic- ular opinions (Breck et al., 2007). Figure 1 shows two examples. This task has been generally solved by supervised learning (˙Irsoy and Cardie, 2014) with the well-established corpus annotated by ex- perts. Almost all previous studies are based on En- glish datasets such as MPQA (Wiebe et al., 2005). By carefully examining this task, we can find that the corpus annotation of opinion expressions is ∗Corresponding author. 武汉是一座英雄的城市 Wuhan is a heroic city 武汉是一座英雄的城市 Wuhan is a heroic city 武汉是一座英雄的城市 Wuhan is a heroic city Annotator-1 Annotator-2 Expert 这几天觉得心里好累 I feel so tired these days 这几天觉得心里好累 I feel so tired these days 这几天觉得心里好累 I feel so tired these days Figure 1: Two examples of opinion expression identi- fication with crowdsourcing and expert annotations in our constructed dataset. The left and right sentences are of positive and negative polarities, respectively. by no means an easy process. It is highly ambigu- ous across different persons. As shown in Figure 1, it is very controversial to define the boundaries of opinion expressions (Wiebe et al., 2005). Ac- tually, this problem is extremely serious for lan- guages such as Chinese, which is based on charac- ters even with no explicit and clearly-defined word boundaries. Thus, Chinese-alike languages will inevitably involve more …

Content truncated to 500 words