Supertagging the Long Tail with Tree-Structured Decoding of Complex Categories

Supertagging the Long Tail with Tree-Structured Decoding of Complex Categories Jakob Prange Nathan Schneider Vivek Srikumar Georgetown University University of Utah {jp1724, nathan.schneider}@georgetown.edu svivek@cs.utah.edu Abstract Although current CCG supertaggers achieve high accuracy on the standard WSJ test set, few systems make use of the categories’ internal structure that will drive the syntac- tic derivation during parsing. The tagset is tra- ditionally truncated, discarding the many rare and complex category types in the long tail. However, supertags are themselves trees. Rather than give up on rare tags, we investigate constructive models that account for their in- ternal structure, including novel methods for tree-structured prediction. Our best tagger is capable of recovering a sizeable fraction of the long-tail supertags and even generates CCG categories that have never been seen in train- ing, while approximating the prior state of the art in overall tag accuracy with fewer param- eters. We further investigate how well differ- ent approaches generalize to out-of-domain evaluation sets. 1 Introduction Combinatory Categorial Grammar (CCG; Steedman, 2000) is a strongly lexicalized grammar formalism in which rich syntactic categories at the lexical level impose tight constraints on the constituents that can be formed. Its syntax-semantics interface has been attractive for downstream tasks such as semantic parsing (Artzi et al., 2015) and machine translation (Nˇadejde et al., 2017). Most CCG parsers operate as a pipeline whose first task is ‘supertagging’, i.e., sequence labeling with a large search space of complex ‘supertags’ (Clark and Curran, 2004; Xu et al., 2015; Vaswani et al., 2016, inter alia). The complex categories specify valency information: expected arguments to the right are signaled with forward slashes, and expected arguments to the left with backward slashes. For example, transitive verbs in English (like ‘‘saw’’ in Figure 1a) are tagged (S\NP)/NP to indicate that they expect a subsequent object noun phrase (NP) and a preceding subject NP to form a clause (S). Given the supertags, all that remains to parsing is applying general rules of (binary) combination between adjacent constitu- ents until the entire input is covered. Supertagging thus represents the crux of the overall parsing process. In contrast to the simpler task of part- of-speech tagging, supertaggers are required to resolve most of the syntactic ambiguity in the input. One key challenge of CCG supertagging is that the tagset is large and open-ended to account for combinatorial possibilities of syntactic construc- tions. This results in a heavy-tailed distribution of supertags, which is visualized in Figure 1b; a large proportion of unique supertags are rare or unseen (out-of-vocabulary, OOV) even in a training set as large as the Penn Treebank’s. Previous CCG supertaggers have surrendered in the face of this challenge: They treat categories as a fixed set of opaque labels, rather than modeling their compositional structure. Following Clark (2002), the standard approach is to consider only supertags appearing at least 10 times in the train- ing data, sacrificing the possibility of predicting two thirds of the supertag types in CCGbank. Rare supertags may have little impact on overall …

Content truncated to 500 words