Named Entity Recognition with Stack Residual LSTM and Trainable Bias Decoding

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 566–575, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Named Entity Recognition with Stack Residual LSTM and Trainable Bias Decoding Quan Tran, Andrew MacKinlay and Antonio Jimeno Yepes IBM Research Australia hung.tran@monash.edu, admackin@au1.ibm.com, ayepes@au1.ibm.com Abstract Recurrent Neural Network models are the state-of-the-art for Named Entity Recog- nition (NER). We present two innovations to improve the performance of these mod- els. The first innovation is the intro- duction of residual connections between the Stacked Recurrent Neural Network model to address the degradation prob- lem of deep neural networks. The sec- ond innovation is a bias decoding mech- anism that allows the trained system to adapt to non-differentiable and externally computed objectives, such as the entity- based F-measure. Our work improves the state-of-the-art results for both Span- ish and English languages on the standard train/development/test split of the CoNLL 2003 Shared Task NER dataset. 1 Introduction In Natural Language Processing, the term “Named Entity” refers to special information units such as people, organizations, location names, numerical expression (Nadeau and Sekine, 2007). Identify- ing the references to these special entities in text is a crucial step toward Language Understanding. Thus, there have been many works on these areas. Some of the early systems employed hand- crafted rules (Rau, 1991; Sekine and Nobata, 2004), however, the vast majority of current sys- tems rely on machine learning models (Nadeau and Sekine, 2007) such as Conditional Random Field (CRF) (McCallum and Li, 2003), Hidden Markov Model (HMM) (Bikel et al., 1997) and Support Vector Machine (SVM) (Asahara and Matsumoto, 2003). Although the traditional ma- chine learning models do not rely on manual rules, they require a manual feature engineering process, which is rather expensive and dependent on the domain and language. In recent years, Recurrent Neural Network (RNN) models such as Long-Short-Term-Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Chung et al., 2014) have been very successful in sequence modeling tasks, for example, Language Modeling (Mikolov et al., 2010; Sundermeyer et al., 2012), Machine Translation (Bahdanau et al., 2014) and Dialog Act Classification (Kalchbrenner and Blunsom, 2013; Tran et al., 2017). RNN models can learn from basic components of text (i.e. words and characters). This generalization capability facil- itates the construction of Language Independent NER models (Ma and Hovy, 2016; Lample et al., 2016) that rely on unsupervised feature learning and a small annotated corpus. One simple way of adding representational power to a neural network is layer stacking. A tra- ditional feed forward neural network usually has three fully connected layers: an input layer, a hid- den layer, and an output layer. For a Convolutional Neural Network (CNN) or Recurrent Neural Net- work, the number of stacked layers might be much larger (Amodei et al., 2016). One problem with this stacking scenario is the degraded representa- tion problem (He et al., 2016). The proposed solu- tion for this problem is the residual-identity con- nection (He et al., …

Content truncated to 500 words