Improved CCG Parsing with Semi-supervised Supertagging

Improved CCG Parsing with Semi-supervised Supertagging Mike Lewis School of Informatics University of Edinburgh Edinburgh, EH8 9AB, UK mike.lewis@ed.ac.uk Mark Steedman School of Informatics University of Edinburgh Edinburgh, EH8 9AB, UK steedman@inf.ed.ac.uk Abstract Current supervised parsers are limited by the size of their labelled training data, making improving them with unlabelled data an im- portant goal. We show how a state-of-the- art CCG parser can be enhanced, by pre- dicting lexical categories using unsupervised vector-space embeddings of words. The use of word embeddings enables our model to better generalize from the labelled data, and allows us to accurately assign lexical cate- gories without depending on a POS-tagger. Our approach leads to substantial improve- ments in dependency parsing results over the standard supervised CCG parser when evalu- ated on Wall Street Journal (0.8%), Wikipedia (1.8%) and biomedical (3.4%) text. We com- pare the performance of two recently proposed approaches for classification using a wide va- riety of word embeddings. We also give a de- tailed error analysis demonstrating where us- ing embeddings outperforms traditional fea- ture sets, and showing how including POS fea- tures can decrease accuracy. 1 Introduction Combinatory Categorial Grammar (CCG) is widely used in natural language semantics (Bos, 2008; Kwiatkowski et al., 2010; Krishnamurthy and Mitchell, 2012; Lewis and Steedman, 2013a; Lewis and Steedman, 2013b; Kwiatkowski et al., 2013), largely because of its direct linkage of syntax and semantics. However, this connection means that performance on semantic applications is highly de- pendent on the quality of the syntactic parse. Al- though CCG parsers perform at state-of-the-art lev- els (Rimell et al., 2009; Nivre et al., 2010), full- sentence accuracy is just 25.6% on Wikipedia text, which gives a low upper bound on logical inference approaches to question-answering and textual entail- ment. Supertags are rich lexical categories that go be- yond POS tags by encoding information about predicate-argument structure. Supertagging is “al- most parsing”, and is used by parsers based on strongly lexicalized formalisms such as CCG and TAG to improve accuracy and efficiency, by dele- gating many of the parsing decisions to finite-state models (Bangalore and Joshi, 1999). A disadvan- tage of this approach is that larger sets of lexical categories mean increased sparsity, decreasing tag- ging accuracy. As large amounts of labelled data are unlikely to be made available, recent work has ex- plored using unlabelled data to improve parser lex- icons (Thomforde and Steedman, 2011; Deoskar et al., 2011; Deoskar et al., 2014). However, existing work has failed to improve the overall accuracy of state-of-the-art supervised parsers in-domain. Another strand of recent work has explored us- ing unsupervised word embeddings as features in supervised models (Turian et al., 2010; Collobert et al., 2011b), largely motivated as a simpler and more general alternative to standard feature sets. We apply similar techniques to CCG supertagging, hypothe- sising that words which are close in the embedding space will have similar supertags. Most existing work has focused on flat tagging tasks, and has not produced state-of-the-art results on structured pre- …

Content truncated to 500 words