Deconstructing Supertagging into Multi-Task Sequence Prediction

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 12–21 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 12 Deconstructing Supertagging into Multi-task Sequence Prediction Zhenqi Zhu School of Computing Science Simon Fraser University Burnaby, BC , Canada zhenqiz@sfu.ca Anoop Sarkar School of Computing Science Simon Fraser University Burnaby, BC , Canada anoop@sfu.ca Abstract Supertagging is a sequence prediction task where each word is assigned a piece of com- plex syntactic structure called a supertag. We provide a novel approach to multi-task learn- ing for Tree Adjoining Grammar (TAG) su- pertagging by deconstructing these complex supertags in order to define a set of re- lated but auxiliary sequence prediction tasks. Our multi-task prediction framework is trained over the exactly same training data used to train the original supertagger where each aux- iliary task provides an alternative view on the original prediction task. Our experimental re- sults show that our multi-task approach signifi- cantly improves TAG supertagging with a new state-of-the-art accuracy score of 91.39% on the Penn treebank supertagging dataset. 1 Introduction A treebank for lexicalized tree-adjoining grammar (TAG) (Joshi and Schabes, 1997) consists of an- notated sentences where each word is provided a complex tree structure called a supertag and the overall parse of the sentence combines these su- pertags into a parse tree. Supertagging is a task that learns a sequence prediction task from this an- notated data and is able to then assign the most likely sequence of supertags to an input sequence of words (Bangalore and Joshi, 1999). Once the right supertag is assigned then parsing is a much easier task and may not even be needed for many applications where information about syntax is needed but a full parse is unnecessary. Supertagging has been shown to be useful for both Tree Adjoining Grammar (TAG) (Banga- lore and Joshi, 1999) and combinatory catego- rial grammar (CCG) (Hockenmaier and Steedman, 2007) parsing. In this paper we aim to improve the state-of-the-art for the task of learning a TAG supertagger from an annotated treebank (Kasai et al., 2018). We observe that supertag predic- tion does not take full advantage of the complex structural information contained within each su- pertag. Neural models have been used to learn em- beddings over these supertags and thereby share weights among similar supertags. Friedman et al. (2017) provide tree-structured neural models over supertags which can learn interesting relationships between supertags but the approach does not lead to higher supertagging accuracy. Our main contribution is to provide several novel ways to deconstruct supertags to create mul- tiple alternative auxiliary tasks, which we then combine using a multi-task prediction framework and we show that this can lead to a significant im- provement in supertagging accuracy. Multi-task learning (MTL) (Caruana, 1997) learns multiple heterogenous tasks in parallel with a shared representation so that what is learned for one task can be shared for another task. In most cases the improvement is due to weight sharing be- tween different tasks (Collobert and …

Content truncated to 500 words