Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8612–8621 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8612 Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation Ivan Provilkov Yandex Research, Moscow Moscow Institute of Physics and Technology iv-provilkov@yandex-team.ru Andrey Malinin Yandex Research, Moscow HSE University, Moscow am969@yandex-team.ru Abstract Neural Machine Translation (NMT) is known to suffer from a beam-search problem: after a certain point, increasing beam size causes an overall drop in translation quality. This effect is especially pronounced for long sen- tences. While much work was done analyzing this phenomenon, primarily for autoregressive NMT models, there is still no consensus on its underlying cause. In this work, we analyze er- rors that cause major quality degradation with large beams in NMT and Automatic Speech Recognition (ASR). We show that a factor that strongly contributes to the quality degradation with large beams is dataset length-bias - NMT datasets are strongly biased towards short sen- tences. To mitigate this issue, we propose a new data augmentation technique – Multi- Sentence Resampling (MSR). This technique extends the training examples by concatenat- ing several sentences from the original dataset to make a long training example. We demon- strate that MSR significantly reduces degrada- tion with growing beam size and improves fi- nal translation quality on the IWSTL15 En-Vi, IWSTL17 En-Fr, and WMT14 En-De datasets. 1 Introduction In this work, we address the beam-search prob- lem in Neural Machine Translation (Koehn and Knowles, 2017). Beam Search is the standard hy- pothesis search method for autoregressive sequence generation. Large beams provide more probable hypotheses than small beams; however, the overall translation quality drops with growing beam size after a certain point. This effect is especially strong for long sentences, connecting with the fact that NMT models are biased to giving high probabil- ities to short hypotheses. Stahlberg and Byrne (2019) showed that exact search by likelihood for neural machine translation finds empty string as the optimal hypothesis in more than 50% of cases. One of the most famous methods to mitigate quality degradation with growing beam size is length normalization (Bahdanau et al., 2016; Wu et al., 2016). This technique normalizes log- likelihoods of a hypothesis in beam search by its length, thus promoting long hypotheses. Other methods examine adding a reward for each token’s score during the decoding process (Yang et al., 2018; Murray and Chiang, 2018). While the beam-search problem has been ex- tensively studied (Sountsov and Sarawagi, 2016; Murray and Chiang, 2018; Kumar and Sarawagi, 2019; Meister et al., 2020; Eikema and Aziz, 2020; Yang et al., 2020; Wang and Sennrich, 2020) there is still no consensus on the underlying reason for such model behavior. Furthermore, prior work has investigated this problem primarily for NMT mod- els, giving little attention to other domains that are also known to suffer from it, such as Auto- matic Speech Recognition (ASR) (Chorowski and Jaitly, 2017; Zhou et al., 2020). Murray and Chi- ang (2018) noticed …

Content truncated to 500 words