Smoothing and Shrinking the Sparse Seq2Seq Search Space

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642–2654 June 6–11, 2021. ©2021 Association for Computational Linguistics 2642 Smoothing and Shrinking the Sparse Seq2Seq Search Space Ben Peters† and Andr´e F. T. Martins†‡∗ †Instituto de Telecomunicac¸˜oes, Instituto Superior T´ecnico, Lisbon, Portugal ‡LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal ∗Unbabel, Lisbon, Portugal benzurdopeters@gmail.com, andre.t.martins@tecnico.ulisboa.pt Abstract Current sequence-to-sequence models are trained to minimize cross-entropy and use soft- max to compute the locally normalized prob- abilities over target sequences. While this setup has led to strong results in a variety of tasks, one unsatisfying aspect is its length bias: models give high scores to short, inadequate hypotheses and often make the empty string the argmax—the so-called cat got your tongue problem. Recently proposed entmax-based sparse sequence-to-sequence models present a possible solution, since they can shrink the search space by assigning zero probability to bad hypotheses, but their ability to handle word-level tasks with transformers has never been tested. In this work, we show that entmax-based models effectively solve the cat got your tongue problem, removing a ma- jor source of model error for neural machine translation. In addition, we generalize label smoothing, a critical regularization technique, to the broader family of Fenchel-Young losses, which includes both cross-entropy and the ent- max losses. Our resulting label-smoothed ent- max loss models set a new state of the art on multilingual grapheme-to-phoneme conver- sion and deliver improvements and better cali- bration properties on cross-lingual morpholog- ical inflection and machine translation for 7 language pairs. 1 Introduction Sequence-to-sequence models (seq2seq: Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) have become a powerful and flexible tool for a variety of NLP tasks, including machine transla- tion (MT), morphological inflection (MI; Faruqui et al., 2016), and grapheme-to-phoneme conver- sion (G2P; Yao and Zweig, 2015). These models often perform well, but they have a bias that fa- vors short hypotheses. This bias is problematic: it has been pointed out as the cause (Koehn and Knowles, 2017; Yang et al., 2018; Murray and Chi- ang, 2018) of the beam search curse, in which increasing the width of beam search actually de- creases performance on neural machine translation (NMT). Further illustrating the severity of the prob- lem, Stahlberg and Byrne (2019) showed that the highest-scoring target sequence in NMT is often the empty string, a phenomenon they dubbed the cat got your tongue problem. These results are undesirable because they show that NMT models’ performance depends on the search errors induced by a narrow beam. It would be preferable for mod- els to assign higher scores to good translations than to bad ones, rather than to depend on search errors to make up for model errors. The most common way to alleviate this short- coming is by altering the decoding objective (Wu et al., 2016; He et al., 2016; Yang et al., 2018; Meister et al., 2020a), but this does not address the underlying problem: the model …

Content truncated to 500 words