Globally Normalized Transition-Based Neural Networks

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2442–2452, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Globally Normalized Transition-Based Neural Networks Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov and Michael Collins∗ Google Inc New York, NY {andor,chrisalberti,djweiss,severyn,apresta,kuzman,slav,mjcollins}@google.com Abstract We introduce a globally normalized transition-based neural network model that achieves state-of-the-art part-of- speech tagging, dependency parsing and sentence compression results. Our model is a simple feed-forward neural network that operates on a task-specific transition system, yet achieves comparable or better accuracies than recurrent models. We dis- cuss the importance of global as opposed to local normalization: a key insight is that the label bias problem implies that globally normalized models can be strictly more expressive than locally normalized models. 1 Introduction Neural network approaches have taken the field of natural language processing (NLP) by storm. In particular, variants of long short-term mem- ory (LSTM) networks (Hochreiter and Schmidhu- ber, 1997) have produced impressive results on some of the classic NLP tasks such as part-of- speech tagging (Ling et al., 2015), syntactic pars- ing (Vinyals et al., 2015) and semantic role label- ing (Zhou and Xu, 2015). One might speculate that it is the recurrent nature of these models that enables these results. In this work we demonstrate that simple feed- forward networks without any recurrence can achieve comparable or better accuracies than LSTMs, as long as they are globally normalized. Our model, described in detail in Section 2, uses a transition system (Nivre, 2006) and feature em- beddings as introduced by Chen and Manning (2014). We do not use any recurrence, but per- form beam search for maintaining multiple hy- ∗On leave from Columbia University. potheses and introduce global normalization with a conditional random field (CRF) objective (Bot- tou et al., 1997; Le Cun et al., 1998; Lafferty et al., 2001; Collobert et al., 2011) to overcome the label bias problem that locally normalized mod- els suffer from. Since we use beam inference, we approximate the partition function by summing over the elements in the beam, and use early up- dates (Collins and Roark, 2004; Zhou et al., 2015). We compute gradients based on this approximate global normalization and perform full backprop- agation training of all neural network parameters based on the CRF loss. In Section 3 we revisit the label bias problem and the implication that globally normalized mod- els are strictly more expressive than locally nor- malized models. Lookahead features can par- tially mitigate this discrepancy, but cannot fully compensate for it—a point to which we return later. To empirically demonstrate the effective- ness of global normalization, we evaluate our model on part-of-speech tagging, syntactic de- pendency parsing and sentence compression (Sec- tion 4). Our model achieves state-of-the-art ac- curacy on all of these tasks, matching or outper- forming LSTMs while being significantly faster. In particular for dependency parsing on the Wall Street Journal we achieve the best-ever published unlabeled attachment score of 94.61%. …

Content truncated to 500 words