Expected F-Measure Training for Shift-Reduce Parsing with Recurrent Neural Networks

Proceedings of NAACL-HLT 2016, pages 210–220, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Expected F-Measure Training for Shift-Reduce Parsing with Recurrent Neural Networks Wenduan Xu † Michael Auli ‡ Stephen Clark † †Computer Laboratory, Cambridge University ‡Facebook AI Research wx217@cam.ac.uk, michaelauli@fb.com, sc609@cam.ac.uk Abstract We present expected F-measure training for shift-reduce parsing with RNNs, which en- ables the learning of a global parsing model optimized for sentence-level F1. We apply the model to CCG parsing, where it improves over a strong greedy RNN baseline, by 1.47% F1, yielding state-of-the-art results for shift- reduce CCG parsing. 1 Introduction Shift-reduce parsing is a popular parsing paradigm, one reason being the potential for fast parsers based on the linear number of parsing actions needed to an- alyze a sentence (Nivre and Scholz, 2004; Sagae and Lavie, 2006; Zhang and Clark, 2011; Goldberg et al., 2013; Zhu et al., 2013; Xu et al., 2014). Recent work has shown that by combining distributed rep- resentations and neural network models (Chen and Manning, 2014), accurate and efficient shift-reduce parsing models can be obtained with little feature engineering, largely alleviating the feature sparsity problem of linear models. In practice, the most common objective for opti- mizing neural network shift-reduce parsing models is maximum likelihood. In the greedy search set- ting, the log-likelihood of each target action is max- imized during training, and the most likely action is committed to at each step of the parsing process dur- ing inference (Chen and Manning, 2014; Dyer et al., 2015). In the beam search setting, Zhou et al. (2015) show that sentence-level likelihood, together with contrastive learning (Hinton, 2002), can be used to derive a global model which incorporates beam search at both training and inference time (Zhang and Clark, 2008), giving significant accuracy gains over a fully greedy model. However, despite the ef- fectiveness of optimizing likelihood, it is often de- sirable to directly optimize for task-specific metrics, which often leads to higher accuracies for a variety of models and applications (Goodman, 1996; Och, 2003; Smith and Eisner, 2006; Rosti et al., 2010; Auli and Lopez, 2011; He and Deng, 2012; Auli et al., 2014; Auli and Gao, 2014; Gao et al., 2014). In this paper, we present a global neural net- work parsing model, optimized for a task-specific loss based on expected F-measure. The model natu- rally incorporates beam search during training, and is globally optimized, to learn shift-reduce action se- quences that lead to parses with high expected F- scores. In contrast to Auli and Lopez (2011), who optimize a CCG parser for F-measure via softmax- margin (Gimpel and Smith, 2010), we directly op- timize an expected F-measure objective, derivable from only a set of shift-reduce action sequences and sentence-level F-scores. More generally, our method can be seen as an alternative approach for training a neural beam search parsing model (Watanabe and Sumita, 2015; Weiss et al., 2015; Zhou et al., 2015), combining the benefits of global learning and task- specific optimization. We also …

Content truncated to 500 words