Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3442–3448 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3442 Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention Yanzeng Li, Bowen Yu, Mengge Xue, Tingwen Liu∗ Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences {liyanzeng, yubowen, xuemengge, liutingwen}@iie.ac.cn Abstract Most Chinese pre-trained models take charac- ter as the basic unit and learn representation according to character’s external contexts, ig- noring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word- aligned attention to exploit explicit word in- formation, which is complementary to vari- ous character-based Chinese pre-trained lan- guage models. Specifically, we devise a pool- ing mechanism to align the character-level at- tention to the word level and propose to alle- viate the potential issue of segmentation error propagation by multi-source information fu- sion. As a result, word and character informa- tion are explicitly integrated at the fine-tuning procedure. Experimental results on five Chi- nese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm. 1 Introduction Pre-trained language Models (PLM) such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), ERNIE (Sun et al., 2019), BERT-wwm (Cui et al., 2019) and XLNet (Yang et al., 2019) have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classifica- tion (Pang et al., 2002), natural language infer- ence (Bowman et al., 2015), named entity recogni- tion (Sang and De Meulder, 2003) and so on. Generally, most popular PLMs prefer to use at- tention mechanism (Vaswani et al., 2017) to rep- resent the natural language, such as word-to-word self-attention for English. Unlike English, in Chi- nese, words are not separated by explicit delimiters. Since without word boundaries information, it is ∗Corresponding author intuitive to model characters in Chinese tasks di- rectly. However, in most cases, the semantic of a single Chinese character is ambiguous. For exam- ple, the character “拍” in word “球拍 (bat)” and “拍卖 (auction)” has entirely different meanings. Moreover, several recent works have demonstrated that considering the word segmentation informa- tion can lead to better language understanding, and accordingly benefits various Chinese tasks (Wang et al., 2017; Li et al., 2018; Zhang and Yang, 2018; Gui et al., 2019; Mengge et al., 2019). All these factors motivate us to expand the character-level attention mechanism in Chinese PLMs to represent the semantics of words 1. To this end, there are two main challenges. (1) How to seamlessly integrate the segmentation information into character-based attention module of PLM is an important problem. (2) Gold-standard segmen- tation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by Chinese word segmentation (CWS) tools (Li et al., 2019) is another challenge. In this paper, we propose a new architec- ture, named Multi-source Word Aligned …

Content truncated to 500 words