An Empirical Investigation of Beam-Aware Training in Supertagging

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4534–4542 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4534 An Empirical Investigation of Beam-Aware Training in Supertagging Renato Negrinho1 Matthew R. Gormley1 Carnegie Mellon University1, MSR Montreal2 {negrinho,mgormley,ggordon}@cs.cmu.edu Geoffrey J. Gordon1,2 Abstract Structured prediction is often approached by training a locally normalized model with max- imum likelihood and decoding approximately with beam search. This approach leads to mismatches as, during training, the model is not exposed to its mistakes and does not use beam search. Beam-aware training aims to ad- dress these problems, but unfortunately, it is not yet widely used due to a lack of under- standing about how it impacts performance, when it is most useful, and whether it is sta- ble. Recently, Negrinho et al. (2018) pro- posed a meta-algorithm that captures beam- aware training algorithms and suggests new ones, but unfortunately did not provide empir- ical results. In this paper, we begin an em- pirical investigation: we train the supertag- ging model of Vaswani et al. (2016) and a simpler model with instantiations of the meta- algorithm. We explore the influence of various design choices and make recommendations for choosing them. We observe that beam-aware training improves performance for both mod- els, with large improvements for the simpler model which must effectively manage uncer- tainty during decoding. Our results suggest that a model must be learned with search to maximize its effectiveness. 1 Introduction Structured prediction often relies on models that train on maximum likelihood and use beam search for approximate decoding. This procedure leads to two significant mismatches between the training and testing settings: the model is trained on oracle trajectories and therefore does not learn about its own mistakes; the model is trained without beam search and therefore does not learn how to use the beam effectively for search. Previous algorithms have addressed one or the other of these mismatches. For example, DAgger (Ross et al., 2011) and scheduled sam- pling (Bengio et al., 2015) use the learned model to visit non-oracle states at training time, but do not use beam search (i.e., they keep a single hy- pothesis). Early update (Collins and Roark, 2004), LaSO (Daum´e and Marcu, 2005), and BSO (Wise- man and Rush, 2016) are trained with beam search, but do not expose the model to beams without a gold hypothesis (i.e., they either stop or reset to beams with a gold hypothesis). Recently, Negrinho et al. (2018) proposed a meta-algorithm that instantiates beam-aware al- gorithms as a result of choices for the surrogate loss (i.e., which training loss to incur at each vis- ited beam) and data collection strategy (i.e., which beams to visit during training). A specific instanti- ation of their meta-algorithm addresses both mis- matches by relying on an insight on how to induce training losses for beams without the gold hypothe- sis: for any beam, its lowest cost neighbor should be scored sufficiently high to be kept in the suc- cessor beam. To induce …

Content truncated to 500 words