Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1744–1753, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser Adhiguna Kuncoro♠ Miguel Ballesteros♦ Lingpeng Kong♠ Chris Dyer♠♣ Noah A. Smith♥ ♠School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA ♦NLP Group, Pompeu Fabra University, Barcelona, Spain ♣Google DeepMind, London, UK ♥Computer Science & Engineering, University of Washington, Seattle, WA, USA {akuncoro,cdyer,lingpenk}@cs.cmu.edu miguel.ballesteros@upf.edu, nasmith@cs.washington.edu Abstract We introduce two first-order graph-based de- pendency parsers achieving a new state of the art. The first is a consensus parser built from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. We cast this approach as minimum Bayes risk decoding (under the Hamming cost) and argue that weaker con- sensus within the ensemble is a useful signal of difficulty or ambiguity. The second parser is a “distillation” of the ensemble into a sin- gle model. We train the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble un- certainty estimates for each possible attach- ment, thereby avoiding the intractable cross- entropy computations required by applying standard distillation objectives to problems with structured outputs. The first-order distil- lation parser matches or surpasses the state of the art on English, Chinese, and German. 1 Introduction Neural network dependency parsers achieve state of the art performance (Dyer et al., 2015; Weiss et al., 2015; Andor et al., 2016), but training them in- volves gradient descent on non-convex objectives, which is unstable with respect to initial parameter values. For some tasks, an ensemble of neural net- works from different random initializations has been found to improve performance over individual mod- els (Sutskever et al., 2014; Vinyals et al., 2015, in- ter alia). In §3, we apply this idea to build a first- order graph-based (FOG) ensemble parser (Sagae and Lavie, 2006) that seeks consensus among 20 randomly-initialized stack LSTM parsers (Dyer et al., 2015), achieving nearly the best-reported per- formance on the standard Penn Treebank Stanford dependencies task (94.51 UAS, 92.70 LAS). We give a probabilistic interpretation to the en- semble parser (with a minor modification), viewing it as an instance of minimum Bayes risk inference. We propose that disagreements among the ensem- ble’s members may be taken as a signal that an at- tachment decision is difficult or ambiguous. Ensemble parsing is not a practical solution, how- ever, since an ensemble of N parsers requires N times as much computation, plus the runtime of find- ing consensus. We address this issue in §5 by distill- ing the ensemble into a single FOG parser with dis- criminative training by defining a new cost function, inspired by the notion of “soft targets” (Hinton et al., 2015). The essential idea is to derive the cost of each possible attachment from the ensemble’s division of votes, and use this cost in discriminative learning. The application of distilliation to structured predic- tion is, to our knowledge, …

Content truncated to 500 words