Porous Lattice Transformer Encoder for Chinese NER

Proceedings of the 28th International Conference on Computational Linguistics, pages 3831–3841 Barcelona, Spain (Online), December 8-13, 2020 3831 Porous Lattice Transformer Encoder for Chinese NER Mengge Xue1,2, Bowen Yu1,2, Tingwen Liu1,2∗, Yue Zhang3 Erli Meng4 and Bin Wang4 1Institute of Information Engineering, Chinese Academy of Sciences, China 2School of Cyber Security, University of Chinese Academy of Sciences, China 3School of Engineering, Westlake University, Hangzhou, China 4Xiaomi AI Lab, Xiaomi Inc., Beijing, China {xuemengge,yubowen,liutingwen}@iie.ac.cn yue.zhang@wias.org.cn {wangbin11,mengerli}@xiaomi.com Abstract Incorporating lexicons into character-level Chinese NER by lattices is proven effective to exploit rich word boundary information. Previous work has extended RNNs to consume lattice inputs and achieved great success. However, due to the DAG structure and the inherently unidirectional sequential nature, this method precludes batched computation and sufficient semantic interaction. In this paper, we propose PLTE, an extension of transformer encoder that is tailored for Chinese NER, which models all the characters and matched lexical words in parallel with batch process- ing. PLTE augments self-attention with positional relation representations to incorporate lattice structure. It also introduces a porous mechanism to augment localness modeling and maintain the strength of capturing the rich long-term dependencies. Experimental results show that PLTE performs up to 11.4 times faster than state-of-the-art methods while realizing better performance. We also demonstrate that using BERT representations further substantially boosts the performance and brings out the best in PLTE. 1 Introduction Named Entity Recognition (NER) is a fundamental task in natural language processing (NLP), which aims to automatically discover named entities and identify their corresponding categories from plain text. NLP tasks such as information retrieval (Berger and Lafferty, 2017), relation extraction (Yu et al., 2019) and entity linking (Xue et al., 2019) require the NER as one of their preprocessing components. Recent studies show that English NER models have achieved improved performance by integrating character information into word representations based on sequence labeling. Different from English NER, East Asian languages including Chinese are written without explicit word boundary. One intuitive way to solve this problem is to segment the input sentences into words first, and then to apply word sequence labeling (Yang et al., 2016; He and Sun, 2017a). However, such methods suffer from error propagation between these two subtasks. To overcome this limitation, efforts have been devoted to incorporating word information by leveraging lexicon features and gazetteers (Peng and Dredze, 2015; Cao et al., 2018; Wu et al., 2019; Lin et al., 2019). As recent state-of-the-art (SOTA) lattice-based method, Zhang and Yang (2018) integrated matched lexical words information into character sequence with a directed acyclic graph (DAG) structure using lattice LSTM. While obtaining promising results, this model faces two challenges. First, as an extension to the non-parallelizable sequential LSTM to a DAG structured model, lattice LSTM is restricted to preprocess one character at a time, which can make it infeasibly to deploy. Second, due to the inherently unidirectional sequential nature, lattice LSTM fails to incorporate the word-level semantics into the representation of the characters except for the last character in each …

Content truncated to 500 words