Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1081–1087, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1081 Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation Pei Zhang, Boxing Chen, Niyu Ge, Kai Fan∗ Alibaba Group Inc. {xiaoyi.zp,boxing.cbx,niyu.ge,k.fan}@alibaba-inc.com Abstract Many document-level neural machine transla- tion (NMT) systems have explored the utility of context-aware architecture, usually requir- ing an increasing number of parameters and computational complexity. However, few at- tention is paid to the baseline model. In this pa- per, we research extensively the pros and cons of the standard transformer in document-level translation, and find that the auto-regressive property can simultaneously bring both the ad- vantage of the consistency and the disadvan- tage of error accumulation. Therefore, we propose a surprisingly simple long-short term masking self-attention on top of the standard transformer to both effectively capture the long-range dependence and reduce the propa- gation of errors. We examine our approach on the two publicly available document-level datasets. We can achieve a strong result in BLEU and capture discourse phenomena. 1 Introduction Recent advances in deep learning have led to signif- icant improvement of Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017). Particularly, the performance on the sentence-level translation of both low- and high- resource lan- guage pairs is dramatically improved (Kudugunta et al., 2019; Lample et al., 2018; Lample and Con- neau, 2019). However, when translating text with long-range dependencies, such as in conversations or documents, the original mode of translating one sentence at a time ignores the discourse phenom- ena (Voita et al., 2019a,b), introducing undesirable behaviors such as inconsistent pronouns across dif- ferent translated sentences. Document-level NMT, as a more realistic transla- tion task in these scenarios, has been systematically ∗corresponding author. investigated in the machine translation community. Most literatures focused on looking back a fixed number of previous source or target sentences as the document-level context (Tu et al., 2018; Voita et al., 2018; Zhang et al., 2018; Miculicich et al., 2018; Voita et al., 2019a,b). Some latest works innovatively attempted to either get the most out of the entire document context or dynamically select the suitable context (Maruf and Haffari, 2018; Yang et al., 2019a; Maruf et al., 2019; Jiang et al., 2019). Because of the scarcity of document training data, the benefit gained from such an approach, as re- flected in BLEU, is usually limited. We therefore elect to pay attention to the context in the previous n sentences only where n is a small number and usually does not cover the entire document. Almost all of the latest studies chose the standard transformer model as their baseline which trans- lates each sentence in the document with the model trained on the sentence-level data. The cohesion and consistency are in general poor. A more rea- sonable baseline is to train the transformer with the context prepended, and this modification could be simply …

Content truncated to 500 words