OptSLA: an Optimization-Based Approach for Sequential Label Aggregation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1335–1340 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1335 OptSLA: an Optimization-Based Approach for Sequential Label Aggregation Nasim Sabetpour, Adithya Kulkarni, and Qi Li Department of Computer Science, Iowa State University {nasim, aditkulk, qli}@iastate.edu Abstract The need for the annotated training dataset on which data-hungry machine learning algo- rithms feed has increased dramatically with ad- vanced acclaim of machine learning applica- tions. To annotate the data, people with do- main expertise are needed, but they are sel- dom available and expensive to hire. This has lead to the thriving of crowdsourcing platforms such as Amazon Mechanical Turk (AMT). However, the annotations provided by one worker cannot be used directly to train the model due to the lack of expertise. Ex- isting literature in annotation aggregation fo- cuses on binary and multi-choice problems. In contrast, little work has been done on complex tasks such as sequence labeling with imbal- anced classes, a ubiquitous task in Natural Lan- guage Processing (NLP), and Bio-Informatics. We propose OPTSLA, an Optimization-based Sequential Label Aggregation method, that jointly considers the characteristics of sequen- tial labeling tasks, workers reliabilities, and advanced deep learning techniques to conquer the challenge. We evaluate our model on crowdsourced data for named entity recogni- tion task. Our results show that the proposed OPTSLA outperforms the state-of-the-art ag- gregation methods, and the results are easier to interpret. 1 Introduction Crowdsourcing (Howe, 2008) is a popular platform to annotate massive corpora inexpensively. It has bred lots of interest in machine learning and deep learning tasks. However, when workers provide annotations, the results may be noisier comparing with labels provided by experts. Thus, it becomes essential to conduct truth inference from the noisy annotations. One common annotation aggregation approach is Majority Voting (MV) (Lam and Suen, 1997), in which annotation with the highest number of occur- rences is deemed as truth. Another naive approach is to regard an annotation as correct if a certain number of workers provide the same annotation. The concern with these methods is that they as- sume all workers are of the same quality, which is usually invalid in practice. In this paper, we study the annotation aggregation problem for sequential labeling tasks, a common NLP task. Many existing crowdsourcing label aggregation methods may suffer from performance loss because they assume that data instances are independent (Zheng et al., 2017). New approaches are recently proposed to handle the particular characteristics of sequential labeling tasks, where tokens in one sentence have complex dependencies (Rodrigues et al., 2014; Simpson and Gurevych, 2019; Nguyen et al., 2017). In this line of approaches, proba- bilistic models are adopted to model the workers’ labeling behavior and to model the dependencies between adjacent tokens. There are some draw- backs to the probabilistic models. First, they have strong statistical assumptions when modeling the sequence annotations, limiting the flexibility of the models. Second, these models need to infer com- plex parameters, making it hard to interpret …

Content truncated to 500 words