More Embeddings, Better Sequence Labelers?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3992–4006 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3992 More Embeddings, Better Sequence Labelers? Xinyu Wang⋄‡, Yong Jiang†∗, Nguyen Bach†, Tao Wang†, Zhongqiang Huang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University ⋄Shanghai Engineering Research Center of Intelligent Vision and Imaging ⋄University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {wangxy1,tukw}@shanghaitech.edu.cn {yongjiang.jy,nguyen.bach}@alibaba-inc.com {leeo.wangt,z.huang,f.huang}@alibaba-inc.com Abstract Recent work proposes a family of contex- tual embeddings that significantly improves the accuracy of sequence labelers over non- contextual embeddings. However, there is no definite conclusion on whether we can build better sequence labelers by combining differ- ent kinds of embeddings in various settings. In this paper, we conduct extensive experi- ments on 3 tasks over 18 datasets and 8 lan- guages to study the accuracy of sequence la- beling with various embedding concatenations and make three observations: (1) concatenat- ing more embedding variants leads to better ac- curacy in rich-resource and cross-domain set- tings and some conditions of low-resource set- tings; (2) concatenating contextual sub-word embeddings with contextual character embed- dings hurts the accuracy in extremely low- resource settings; (3) based on the conclusion of (1), concatenating additional similar con- textual embeddings cannot lead to further im- provements. We hope these conclusions can help people build stronger sequence labelers in various settings. 1 Introduction In recent years, sequence labelers equipped with contextual embeddings have achieved significant accuracy improvement (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2019; Martin et al., 2019) over approaches that use static non-contextual word embeddings (Mikolov et al., 2013) and character embeddings (Santos and Zadrozny, 2014). Differ- ent types of embeddings have different inductive biases to guide the learning process. However, little work has been done to study how to concatenate these contextual embeddings and non-contextual embeddings to build better sequence labelers in ∗Yong Jiang and Kewei Tu are the corresponding authors. ‡: This work was conducted when Xinyu Wang was interning at Alibaba DAMO Academy. multilingual, low-resource, or cross-domain set- tings over various sequence labeling tasks. In this paper, we empirically investigate the effectiveness of concatenating various kinds of embeddings for multilingual sequence labeling and try to answer the following questions: 1. In rich-resources settings, does combining different kinds of contextual embeddings re- sult in a better sequence labeler? Are non- contextual embeddings helpful when the mod- els are equipped with contextual embeddings? 2. When we train models in low-resource and cross-domain settings, do the conclusions from the rich-resource settings still hold? 3. Can sequence labelers automatically learn the importance of each kind of embeddings when they are concatenated? 2 Model Architecture 2.1 Sequence Labeling We use the BiLSTM structure for all the sequence labeling tasks, which is one of the most popular ap- proaches to sequence labeling (Huang et al., 2015; Ma and Hovy, 2016). Given a n word sentence x = {x1, · · · , xn} and L kinds of embeddings, we feed the sentence …

Content truncated to 500 words