An Empirical Exploration of Skip Connections for Sequential Tagging

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 203–212, Osaka, Japan, December 11-17 2016. An Empirical Exploration of Skip Connections for Sequential Tagging Huijia Wu1,3, Jiajun Zhang1,3, and Chengqing Zong1,2,3 1National Laboratory of Pattern Recognition, Institute of Automation, CAS 2CAS Center for Excellence in Brain Science and Intelligence Technology 3University of Chinese Academy of Sciences {huijia.wu,jjzhang,cqzong}@nlpr.ia.ac.cn Abstract In this paper, we empirically explore the effects of various kinds of skip connections in stacked bidirectional LSTMs for sequential tagging. We investigate three kinds of skip connections con- necting to LSTM cells: (a) skip connections to the gates, (b) skip connections to the internal states and (c) skip connections to the cell outputs. We present comprehensive experiments show- ing that skip connections to cell outputs outperform the remaining two. Furthermore, we observe that using gated identity functions as skip mappings works pretty well. Based on this novel skip connections, we successfully train deep stacked bidirectional LSTM models and obtain state-of- the-art results on CCG supertagging and comparable results on POS tagging. 1 Introduction In natural language processing, sequential tagging mainly refers to the tasks of assigning discrete labels to each token in a sequence. Typical examples include part-of-speech (POS) tagging and combinatory category grammar (CCG) supertagging. A regular feature of sequential tagging is that the input tokens in a sequence cannot be assumed to be independent since the same token in different contexts can be assigned to different tags. Therefore, the classifier should have memories to remember the contexts to make a correct prediction. Bidirectional LSTMs (Graves and Schmidhuber, 2005) become dominant in sequential tagging prob- lems due to the superior performance (Wang et al., 2015; Vaswani et al., 2016; Lample et al., 2016). The horizontal hierarchy of LSTMs with bidirectional processing can remember the long-range depen- dencies without affecting the short-term storage. Although the models have a deep horizontal hierarchy (the depth is the same as the sequence length), the vertical hierarchy is often shallow, which may not be efficient at representing each token. Stacked LSTMs are deep in both directions, but become harder to train due to the feed-forward structure of stacked layers. Skip connections (or shortcut connections) enable unimpeded information flow by adding direct con- nections across different layers (Raiko et al., 2012; Graves, 2013; Hermans and Schrauwen, 2013). How- ever, there is a lack of exploration and analyzing various kinds of skip connections in stacked LSTMs. There are two issues to handle skip connections in stacked LSTMs: One is where to add the skip con- nections, the other is what kind of skip connections should be used to pass the information. To answer the first question, we empirically analyze three positions of LSTM blocks to receive the previous layer’s output. For the second one, we present an identity mapping to receive the previous layer’s outputs. Fur- thermore, following the gate design of LSTM (Hochreiter and Schmidhuber, 1997; Gers et al., 2000) and highway networks (Srivastava et al., 2015a; Srivastava et al., 2015b), …

Content truncated to 500 words