Supertagging With LSTMs

Proceedings of NAACL-HLT 2016, pages 232–237, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Supertagging with LSTMs Ashish Vaswani1, Yonatan Bisk1, Kenji Sagae2, and Ryan Musa3 1University of Southern California, 2Kitt.ai 3University of Illinois at Urbana-Champaign vaswani@usc.edu, ybisk@isi.edu sagae@kitt.ai, ramusa2@illinois.edu Abstract In this paper we present new state-of-the-art performance on CCG supertagging and pars- ing. Our model outperforms existing ap- proaches by an absolute gain of 1.5%. We an- alyze the performance of several neural mod- els and demonstrate that while feed-forward architectures can compete with bidirectional LSTMs on POS tagging, models that encode the complete sentence are necessary for the long range syntactic information encoded in supertags. 1 Introduction Morphosyntactic labels for words are commonly used in a variety of NLP applications. For this rea- son, part-of-speech (POS) tagging and supertagging have drawn significant attention from the commu- nity. Combinatory Categorial Grammar is a lexical- ized grammar formalism that is widely used for syn- tactic and semantic parsing. Supertagging (Clark, 2002; Bangalore and Joshi, 2010) assigns complex syntactic labels to words to enable fast and accurate parsing. The disambiguation of correctly labeling a word with one of over 1,200 CCG labels is dif- ficult compared to choosing on of the 45 POS la- bels in the Penn Treebank (Marcus et al., 1993). In addition to the large label space of CCG supertags, labeling a word correctly depends on knowledge of syntactic phenomena arbitrarily far in the sentence (Hockenmaier and Steedman, 2007). This is be- cause supertags encode highly specific syntactic in- formation (e.g. types and locations of arguments) about a word’s usage in a sentence. In this paper, we show that Bidirectional Long Short-Term Memory recurrent neural networks (bi– LSTMs) (Graves, 2013; Zaremba et al., 2014), which can use information from the entire sentence, are a natural and powerful architecture for CCG su- pertagging. In addition to the bi–LSTM, we create a simple yet novel model that outperforms the pre- vious state-of-the-art RNN model that uses hand- crafted features (Xu et al., 2015) by 1.5%. Con- current to this work (Lewis et al., 2016) introduced a different training methodology for bi-LSTM for supertagging. We provide a detailed analysis of the quality of various LSTM architectures, forward, backward, and bi-directional, shedding light over the ability of the bi–LSTM to exploit rich sentential con- text necessary for performing supertagging. We also show that a baseline feed-forward neural network (NN) architecture significantly outperforms previ- ous feed-forward NN baselines, with slightly fewer features, achieving better accuracy than the RNN model from (Xu et al., 2015). Recently, bi–LSTMs have achieved high accu- racies in a simpler sequence labeling task: part- of-speech tagging (Wang et al., 2015; Ling et al., 2015) on the Penn treebank, with small improve- ments over local models. However, we achieve strong accuracies compared to (Wang et al., 2015) using feed-forward neural network model trained on local context, showing that this task does not require bi–LSTMs. Our strong feed-forward NN baselines show the power of feed-forward NNs for …

Content truncated to 500 words