An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search

Proceedings of NAACL-HLT 2019, pages 1724–1733 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1724 An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search Kartik Goyal School of Computer Science Carnegie Mellon University kartikgo@cs.cmu.edu Chris Dyer Deepmind cdyer@google.com Taylor Berg-Kirkpatrick University of California, San Diego tberg@ucsd.edu Abstract Globally normalized neural sequence models are considered superior to their locally normal- ized equivalents because they may ameliorate the effects of label bias. However, when con- sidering high-capacity neural parametrizations that condition on the whole input sequence, both model classes are theoretically equiva- lent in terms of the distributions they are ca- pable of representing. Thus, the practical ad- vantage of global normalization in the con- text of modern neural methods remains un- clear. In this paper, we attempt to shed light on this problem through an empirical study. We extend an approach for search-aware train- ing via a continuous relaxation of beam search (Goyal et al., 2017b) in order to enable train- ing of globally normalized recurrent sequence models through simple backpropagation. We then use this technique to conduct an empir- ical study of the interaction between global normalization, high-capacity encoders, and search-aware optimization. We observe that in the context of inexact search, globally nor- malized neural models are still more effec- tive than their locally normalized counterparts. Further, since our training approach is sen- sitive to warm-starting with pre-trained mod- els, we also propose a novel initialization strategy based on self-normalization for pre- training globally normalized models. We per- form analysis of our approach on two tasks: CCG supertagging and Machine Translation, and demonstrate the importance of global nor- malization under different conditions while us- ing search-aware training. 1 Introduction Neural encoder-decoder models have been tremendously successful at a variety of NLP tasks, such as machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), parsing (Dyer et al., 2016, 2015), summarization (Rush et al., 2015), dialog generation (Serban et al., 2015), and image captioning (Xu et al., 2015). With these models, the target sequence is generated in a left-to-right step-wise manner with the predictions at every step being conditioned on the input sequence and the whole prediction history. This long-distance memory precludes exact search for the maximally scoring sequence according to the model and therefore, approximate algorithms like greedy search or beam search are necessary in practice during decoding. In this scenario, it is natural to resort to search-aware learning techniques for these models which makes the optimization objective sensitive to any potential errors that could occur due to inexact search in these models. This work focuses on comparison between search-aware locally normalized sequence mod- els that involve projecting the scores of items in the vocabulary onto a probability simplex at each step and globally normalized/unnormalized sequence models that involve scoring sequences without explicit normalization at each step. When conditioned on the the full input sequence and the entire prediction history, both locally normal- …

Content truncated to 500 words