Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter

Lexicon Enhanced Chinese Sequence Labelling Using BERT Adapter Wei Liu1, Xiyan Fu2, Yue Zhang3, Wenming Xiao1 1DAMO Academy, Alibaba Group, China 2College of Computer Science, Nankai University, China 3School of Engineering, Westlake University, China 3Institute of Advanced Technology, Westlake Institute for Advanced Study hezan.lw@alibaba-inc.com, fuxiyan@mail.nankai.edu.cn, yue.zhang@wias.org.cn, wenming.xiaowm@alibaba-inc.com Abstract Lexicon information and pre-trained models, such as BERT, have been combined to explore Chinese sequence labelling tasks due to their respective strength. However, existing meth- ods solely fuse lexicon features via a shal- low and random initialized sequence layer and do not integrate them into the bottom layers of BERT. In this paper, we propose Lexicon Enhanced BERT (LEBERT) for Chinese se- quence labelling, which integrates external lex- icon knowledge into BERT layers directly by a Lexicon Adapter layer. Compared with the existing methods, our model facilitates deep lexicon knowledge fusion at the lower layers of BERT. Experiments on ten Chinese datasets of three tasks including Named Entity Recogni- tion, Word Segmentation, and Part-of-Speech Tagging, show that LEBERT achieves the state- of-the-art results. 1 Introduction Sequence labeling is a classic task in natural lan- guage processing (NLP), which is to assign a label to each unit in a sequence (Jurafsky and Martin, 2009). Many important language processing tasks can be converted into this problem, such as part- of-speech (POS) tagging, named entity recognition (NER) and text chunking. The current state-of- the-art results for sequence labelling have been achieved by neural network approaches (Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016; Gui et al., 2017). Chinese sequence labelling is more challenging due to the lack of explicit word boundaries in Chi- nese sentences. One way of performing Chinese sequence labelling is to perform Chinese word seg- mentation (CWS) first, before applying word se- quence labelling (Sun and Uszkoreit, 2012; Yang et al., 2016). However, it can suffer from the seg- mentation errors propagated from the CWS system !"#$%&'"()" !"#$%&'"()" !! "! "" !" "# !# #! #" ## *+%,'$ -#.)" /$&)")$0) -#.)" 123! (a) Model-level Fusion !"#$%&'"()" !"#$%&'"()" !! "! "" !" "# !# #! #" ## *$&)")$+) ,#-)" ./0! (b) BERT-level Fusion Figure 1: Comparison of fusing lexicon features and BERT at different levels for Chinese sequence labelling. For simplicity, we only show two Transformer layers in BERT and truncate the sentence to three characters. ci denotes the i-th Chinese character, wj denotes the j-th Chinese word. (Zhang and Yang, 2018; Liu et al., 2019). There- fore, some approaches (Cao et al., 2018; Shen et al., 2016) perform Chinese sequence labelling directly at the character level, which have been em- pirically proven to be more effective (Ng and Low, 2004; Liu et al., 2010; Zhang and Yang, 2018). There are two lines of recent work enhancing character-based neural Chinese sequence labelling. The first considers integrating word information into a character-based sequence encoder, so that word features can be explicitly modelled (Zhang and Yang, 2018; Yang et al., 2019; Liu et al., 2019; Ding et al., 2019; Higashiyama et …

Content truncated to 500 words