Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2134–2146 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2134 Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling Ouyu Lan†∗ Xiao Huang†∗ Bill Yuchen Lin† He Jiang† Liyuan Liu‡ Xiang Ren† {olan,huan183,yuchen.lin,jian567,xiangren}@usc.edu, ll2@illinois.edu †Computer Science Department, University of Southern California ‡Computer Science Department, University of Illinois at Urbana-Champaign Abstract Sequence labeling is a fundamental task for a range of natural language processing prob- lems. When used in practice, its performance is largely influenced by the annotation qual- ity and quantity, and meanwhile, obtaining ground truth labels is often costly. In many cases, ground truth labels do not exist, but noisy annotations or annotations from differ- ent domains are accessible. In this paper, we propose a novel framework Consensus Net- work (CONNET) that can be trained on annota- tions from multiple sources (e.g., crowd anno- tation, cross-domain data). It learns individual representation for every source and dynami- cally aggregates source-specific knowledge by a context-aware attention module. Finally, it leads to a model reflecting the agreement (con- sensus) among multiple sources. We evaluate the proposed framework in two practical set- tings of multi-source learning: learning with crowd annotations and unsupervised cross- domain model adaptation. Extensive experi- mental results show that our model achieves significant improvements over existing meth- ods in both settings. We also demonstrate that the method can apply to various tasks and cope with different encoders. 1 1 Introduction Sequence labeling is a general approach en- compassing various natural language process- ing (NLP) tasks including part-of-speech (POS) tagging (Ratnaparkhi, 1996), word segmenta- tion (Low et al., 2005), and named entity recogni- tion (NER) (Nadeau and Sekine, 2007). Typically, existing methods follow the supervised learning paradigm, and require high-quality annotations. While gold standard annotation is expensive and ∗The first two authors contributed equally. 1Our code can be found at https://github.com/ INK-USC/ConNet . time-consuming, imperfect annotations are rela- tively easier to obtain from crowdsourcing (noisy labels) or other domains (out-of-domain). De- spite their low cost, such supervision usually can be obtained from different sources, and it has been shown that multi-source weak supervision has the potential to perform similar to gold anno- tations (Ratner et al., 2016). Specifically, we are interested in two scenar- ios: 1) learning with crowd annotations and 2) unsupervised cross-domain model adaptation. Both situations suffer from imperfect annotations, and benefit from multiple sources. Therefore, the key challenge here is to aggregate multi-source imperfect annotations for learning a model with- out knowing the underlying ground truth label se- quences in the target domain. Our intuition mainly comes from the phe- nomenon that different sources of supervision have different strengths and are more proficient with distinct situations. Therefore they may not keep consistent importance during aggregating su- pervisions, and aggregating multiple sources for a specific input should be a dynamic process that de- pends on the sentence context. To better model this nature, we need to (1) explicitly model the unique traits of different …

Content truncated to 500 words