Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2077–2083, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling Gakuto Kurata IBM Research gakuto@jp.ibm.com Bing Xiang IBM Watson bingxia@us.ibm.com Bowen Zhou IBM Watson zhou@us.ibm.com Mo Yu IBM Watson yum@us.ibm.com Abstract Recurrent Neural Network (RNN) and one of its specific architectures, Long Short-Term Memory (LSTM), have been widely used for sequence labeling. Explicitly modeling out- put label dependencies on top of RNN/LSTM is a widely-studied and effective extension. We propose another extension to incorpo- rate the global information spanning over the whole input sequence. The proposed method, encoder-labeler LSTM, first encodes the whole input sequence into a fixed length vector with the encoder LSTM, and then uses this encoded vector as the initial state of an- other LSTM for sequence labeling. With this method, we can predict the label sequence while taking the whole input sequence in- formation into consideration. In the experi- ments of a slot filling task, which is an es- sential component of natural language under- standing, with using the standard ATIS cor- pus, we achieved the state-of-the-art F1-score of 95.66%. 1 Introduction Natural language understanding (NLU) is an essen- tial component of natural human computer interac- tion and typically consists of identifying the intent of the users (intent classification) and extracting the as- sociated semantic slots (slot filling) (De Mori et al., 2008). We focus on the latter semantic slot filling task in this paper. Slot filling can be framed as a sequential label- ing problem in which the most probable semantic slot labels are estimated for each word of the given word sequence. Slot filling is a traditional task and tremendous efforts have been done, especially since the 1980s when the Defense Advanced Research Program Agency (DARPA) Airline Travel Informa- tion System (ATIS) projects started (Price, 1990). Following the success of deep learning (Hinton et al., 2006; Bengio, 2009), Recurrent Neural Net- work (RNN) (Elman, 1990; Jordan, 1997) and one of its specific architectures, Long Short-Term Mem- ory (LSTM) (Hochreiter and Schmidhuber, 1997), have been widely used since they can capture tem- poral dependencies (Yao et al., 2013; Yao et al., 2014a; Mesnil et al., 2015). The RNN/LSTM-based slot filling has been extended to be combined with explicit modeling of label dependencies (Yao et al., 2014b; Liu and Lane, 2015). In this paper, we extend the LSTM-based slot filling to consider sentence-level information. In the field of machine translation, an encoder-decoder LSTM has been gaining attention (Sutskever et al., 2014), where the encoder LSTM encodes the global information spanning over the whole input sentence in its last hidden state. Inspired by this idea, we pro- pose an encoder-labeler LSTM that leverages the en- coder LSTM for slot filling. First, we encode the in- put sentence into a fixed length vector by the encoder LSTM. Then, we predict the slot label sequence by the labeler LSTM whose hidden state is initialized with …

Content truncated to 500 words