Recurrent Support Vector Machines For Slot Tagging In Spoken Language Understanding

Proceedings of NAACL-HLT 2016, pages 393–399, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Recurrent Support Vector Machines For Slot Tagging In Spoken Language Understanding Yangyang Shi and Kaisheng Yao and Hu Chen and Dong Yu Yi-Cheng Pan and Mei-Yuh Hwang Microsoft {yanshi,kaisheng.yao,huch,dong.yu,ycpan,mehwang}@microsoft.com Abstract We propose recurrent support vector machine (RSVM) for slot tagging. This model is a combi- nation of the recurrent neural network (RNN) and the structured support vector machine. RNN extracts features from the input sequence. The structured support vector machine uses a sequence-level dis- criminative objective function. The proposed model therefore combines the sequence representation ca- pability of an RNN with the sequence-level discrim- inative objective. We have observed new state-of- the-art results on two benchmark datasets and one private dataset. RSVM obtained statistical significant 4% and 2% relative average F1 score improvement on ATIS dataset and Chunking dataset, respectively. Out of eight domains in Cortana live log dataset, RSVM achieved F1 score improvement on seven do- mains. Experiments also show that RSVM signif- icantly speeds up the model training by skipping the weight updating for non-support vector training samples, compared against training using RNN with CRF or minimum cross-entropy objectives. 1 Introduction One of the key tasks in natural language understanding (Hemphill et al., 1990a; He and Young, 2003; De Mori, 2007; Dinarelli et al., 2008; Wang et al., 2005) is slot tag- ging that labels user queries with semantic tags. It is a sequence labeling problem that transcribes a sequence of observations X = [x(1),x(2),...,x(M)] to a sequence of discrete labels Y = [y(1),y(2),...,y(M)]. For example, in the query “show me flights from Seattle to Boston”, the words “Seattle” and “Boston” should be labeled, respec- tively, as the from-city-name slot and the to-city-name slot. Recently recurrent neural networks (RNNs) and their variants achieved state-of-the-art performances on slot tagging tasks (Yao et al., 2013; Yao et al., 2014b; Yao et al., 2014a; Graves, 2012; Shi et al., 2015; Mesnil et al., 2015; Peng and Yao, 2015). One direction to improve the sequence labeling is to strengthen the model memoriza- tion capability by designing dedicated special structures, for example, using long-short-term-memory (LSTM) net- works (Hochreiter and Schmidhuber, 1997; Graves et al., 2013; Yao et al., 2014a), gated RNN and RNN with ex- ternal memory (RNN-em) (Peng and Yao, 2015). The other direction is to optimize the sequence-level discrim- ination criterion. For example, recurrent conditional ran- dom fields (RCRFs) (Yao et al., 2014b) is trained to opti- mize the sequence conditional likelihood rather than min- imizing frame level cross-entropy applied in conventional RNN based sequence labeling (Prez-ortiz et al., 2001; Yao et al., 2013; Mikolov et al., 2010; Shi et al., 2015; Mesnil et al., 2015). In this paper, we propose recurrent support vector ma- chines (RSVMs) to improve the discrimination ability of RNNs. Different from RCRFs and conventional RNNs that in essence apply the multinomial logistic regression on the output layer, RSVMs optimize the sequence-level max-margin training criterion used by structured support vector …

Content truncated to 500 words