Automated Concatenation of Embeddings for Structured Prediction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2643–2660 August 1–6, 2021. ©2021 Association for Computational Linguistics 2643 Automated Concatenation of Embeddings for Structured Prediction Xinyu Wang⋄‡, Yong Jiang†∗, Nguyen Bach†, Tao Wang†, Zhongqiang Huang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {wangxy1,tukw}@shanghaitech.edu.cn, yongjiang.jy@alibaba-inc.com {nguyen.bach,leeo.wangt,z.huang,f.huang}@alibaba-inc.com Abstract Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that bet- ter word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collec- tion of candidate embeddings, and the ever- increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embed- dings (ACE) to automate the process of find- ing better concatenations of embeddings for structured prediction tasks, based on a formu- lation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embed- dings, according to its current belief of the ef- fectiveness of individual embedding types in consideration for a task, and updates the be- lief based on a reward. We follow strategies in reinforcement learning to optimize the pa- rameters of the controller and compute the re- ward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empir- ical results on 6 tasks and 21 datasets show that our approach outperforms strong base- lines and achieves state-of-the-art performance with fine-tuned embeddings in all the evalua- tions.1 1 Introduction Recent developments on pretrained contextualized embeddings have significantly improved the per- formance of structured prediction tasks in natural ∗Yong Jiang and Kewei Tu are the corresponding authors. ‡: This work was conducted when Xinyu Wang was interning at Alibaba DAMO Academy. 1Our code is publicly available at https://github. com/Alibaba-NLP/ACE. language processing. Approaches based on contex- tualized embeddings, such as ELMo (Peters et al., 2018), Flair (Akbik et al., 2018), BERT (Devlin et al., 2019), and XLM-R (Conneau et al., 2020), have been consistently raising the state-of-the-art for various structured prediction tasks. Concur- rently, research has also showed that word represen- tations based on the concatenation of multiple pre- trained contextualized embeddings and traditional non-contextualized embeddings (such as word2vec (Mikolov et al., 2013) and character embeddings (Santos and Zadrozny, 2014)) can further improve performance (Peters et al., 2018; Akbik et al., 2018; Straková et al., 2019; Wang et al., 2020b). Given the ever-increasing number of embedding learn- ing methods that operate on different granularities (e.g., word, subword, or character level) and with different model architectures, choosing the best em- beddings to concatenate for a specific task becomes non-trivial, and exploring …

Content truncated to 500 words