Predicting and Using Target Length in Neural Machine Translation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 389–395 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 389 Predicting and Using Target Length in Neural Machine Translation Zijian Yang Yingbo Gao Weiyue Wang Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {zyang|gao|wwang|ney}@i6.informatik.rwth-aachen.de Abstract Attention-based encoder-decoder models have achieved great success in neural machine trans- lation tasks. However, the lengths of the target sequences are not explicitly predicted in these models. This work proposes length prediction as an auxiliary task and set up a sub-network to obtain the length information from the encoder. Experimental results show that the length pre- diction sub-network brings improvements over the strong baseline system and that the pre- dicted length can be used as an alternative to length normalization during decoding. 1 Introduction In recent years, neural network (NN) models have achieved great improvements in machine transla- tion (MT) tasks. Sutskever et al. (2014) intro- duced the encoder-decoder network, Bahdanau et al. (2015) developed the attention-based archi- tecture, and Vaswani et al. (2017) proposed the transformer model with self-attentions, which de- livers state-of-the-art performances. Despite the success achieved in neural machine translation (NMT), current NMT systems do not model the length of the output explicitly, and thus various length normalization approaches are often used in decoding. Length normalization is a com- mon technique used in the beam search of NMT systems to enable a fair comparison of partial hy- potheses with different lengths. Without any form of length normalization, regular beam searches will prefer shorter hypotheses to longer ones on average, as a negative logarithmic probability is added at each step, resulting in lower (more negative) scores for longer sentences. The simplest way is to nor- malize the score of the current partial hypothesis (ei 1) by its length (|i|): s(ei 1, fJ 1 ) = log p(ei 1|fJ 1 ) |i| (1) where fJ 1 is the source sequence. To use a softer approach, the denominator |i| can also be raised to the power of a number between 0 and 1 or replaced by more complex functions, as proposed in Wu et al. (2016). Moreover, a constant word reward is used in He et al. (2016) as an alternative to length normalization. All of these approaches tackle the length problem in decoding, and all NMT systems use at least one of them to ensure the performance. In addition to investigating various types of length normalization, their rationality is rarely ex- plored. Although length normalization appears to be simple and effective, it is still an additional tech- nique to help a “weak” machine translation model that cannot handle the hypothesis length properly. In this work it is proposed to model the target length using the neural network itself in a multi-task learn- ing way. The estimated length information can ei- ther be implicitly included in the …

Content truncated to 500 words