GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2431–2441 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2431 GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling Yijin Liu1∗, Fandong Meng2, Jinchao Zhang2, Jinan Xu1†, Yufeng Chen1 and Jie Zhou2 1Beijing Jiaotong University, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China adaxry@gmail.com {fandongmeng, dayerzhang, withtomzhou}@tencent.com {jaxu,chenyf}@bjtu.edu.cn Abstract Current state-of-the-art systems for the se- quence labeling tasks are typically based on the family of Recurrent Neural Networks (RNNs). However, the shallow connections between consecutive hidden states of RNNs and insufficient modeling of global informa- tion restrict the potential performance of those models. In this paper, we try to address these issues, and thus propose a Global Context en- hanced Deep Transition architecture for se- quence labeling named GCDT. We deepen the state transition path at each position in a sen- tence, and further assign every token with a global representation learned from the entire sentence. Experiments on two standard se- quence labeling tasks show that, given only training data and the ubiquitous word embed- dings (Glove), our GCDT achieves 91.96 F1 on the CoNLL03 NER task and 95.43 F1 on the CoNLL2000 Chunking task, which outper- forms the best reported results under the same settings. Furthermore, by leveraging BERT as an additional resource, we establish new state- of-the-art results with 93.47 F1 on NER and 97.30 F1 on Chunking 1. 1 Introduction Sequence labeling tasks, including part-of-speech tagging (POS), syntactic chunking and named en- tity recognition (NER), are fundamental and chal- lenging problems of Natural Language Process- ing (NLP). Recently, neural models have become the de-facto standard for high-performance sys- tems. Among various neural networks for se- quence labeling, bi-directional RNNs (BiRNNs), especially BiLSTMs (Hochreiter and Schmidhu- ber, 1997) have become a dominant method on ∗ This work was done when Yijin Liu was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China †Jinan Xu is the corresponding author of the paper. 1Code is available at: https://github.com/Adaxry/GCDT. multiple benchmark datasets (Huang et al., 2015; Chiu and Nichols, 2016; Lample et al., 2016; Pe- ters et al., 2017). However, there are several natural limitations of the BiLSTMs architecture. For example, at each time step, the BiLSTMs consume an incom- ing word and construct a new summary of the past subsequence. This procedure should be highly nonlinear, to allow the hidden states to rapidly adapt to the mutable input while still preserving a useful summary of the past (Pascanu et al., 2014). While in BiLSTMs, even stacked BiLSTMs, the transition depth between consecutive hidden states are inherently shallow. Moreover, global contex- tual information, which has been shown highly useful for model sequence (Zhang et al., 2018), is insufficiently captured at each token position in BiLSTMs. Subsequently, inadequate representa- tions flow into the final prediction layer, which leads to the restricted performance of BiLSTMs. In this paper, we present a global context en- hanced deep transition architecture to eliminate the …

Content truncated to 500 words