Stack-propagation: Improved Representation Learning for Syntax

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1557–1566, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Stack-propagation: Improved Representation Learning for Syntax Yuan Zhang∗ CSAIL, MIT Cambridge, MA 02139, USA yuanzh@csail.mit.edu David Weiss Google Inc New York, NY 10027, USA djweiss@google.com Abstract Traditional syntax models typically lever- age part-of-speech (POS) information by constructing features from hand-tuned templates. We demonstrate that a better approach is to utilize POS tags as a reg- ularizer of learned representations. We propose a simple method for learning a stacked pipeline of models which we call “stack-propagation”. We apply this to de- pendency parsing and tagging, where we use the hidden layer of the tagger network as a representation of the input tokens for the parser. At test time, our parser does not require predicted POS tags. On 19 lan- guages from the Universal Dependencies, our method is 1.3% (absolute) more accu- rate than a state-of-the-art graph-based ap- proach and 2.7% more accurate than the most comparable greedy model. 1 Introduction In recent years, transition-based dependency parsers powered by neural network scoring func- tions have dramatically increased the state-of-the- art in terms of both speed and accuracy (Chen and Manning, 2014; Alberti et al., 2015; Weiss et al., 2015). Similar approaches also achieve state-of- the-art in other NLP tasks, such as constituency parsing (Durrett and Klein, 2015) or semantic role labeling (FitzGerald et al., 2015). approaches all share a common principle:These re- place hand-tuned conjunctions of traditional NLP feature templates with continuous approximations learned by the hidden layer of a feed-forward net- work. ∗Research conducted at Google. However, state-of-the-art dependency parsers depend crucially on the use of predicted part-of- speech (POS) tags. In the pipeline or stacking (Wolpert, 1992) method, these are predicted from an independently trained tagger and used as fea- tures in the parser. However, there are two main disadvantages of a pipeline: (1) errors from the POS tagger cascade into parsing errors, and (2) POS taggers often make mistakes precisely be- cause they cannot take into account the syntactic context of a parse tree. The POS tags may also contain only coarse information, such as when us- ing the universal tagset of Petrov et al. (2011). One approach to solve these issues has been to avoid using POS tags during parsing, e.g. either using semi-supervised clustering instead of POS tags (Koo et al., 2008) or building recurrent repre- sentations of words using neural networks (Dyer et al., 2015; Ballesteros et al., 2015). However, the best accuracy for these approaches is still achieved by running a POS tagger over the data first and combining the predicted POS tags with additional representations. As an alternative, a wide range of prior work has investigated jointly modeling both POS and parse trees (Li et al., 2011; Hatori et al., 2011; Bohnet and Nivre, 2012; Qian and Liu, 2012; Wang and Xue, 2014; Li et al., 2014; Zhang et al., 2015; Alberti et al., 2015). However, these approaches typically require …

Content truncated to 500 words