Generic Mechanism for Reducing Repetitions in Encoder-Decoder Models

Proceedings of Recent Advances in Natural Language Processing, pages 1606–1615 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_180 1606 Generic Mechanism for Reducing Repetitions in Encoder-Decoder Models Ying Zhang1, Hidetaka Kamigaito1, Tatsuya Aoki1, Hiroya Takamura2 and Manabu Okumura1 1Tokyo Institute of Technology 2National Institute of Advanced Industrial Science and Technology (AIST) {zhang,kamigaito,aoki,oku}@lr.pi.titech.ac.jp takamura.hiroya@aist.go.jp Abstract Encoder-decoder models have been commonly used for many tasks such as machine transla- tion and response generation. As previous re- search reported, these models suffer from gen- erating redundant repetition. In this research, we propose a new mechanism for encoder- decoder models that estimates the semantic dif- ference of a source sentence before and af- ter being fed into the encoder-decoder model to capture the consistency between two sides. This mechanism helps reduce repeatedly gen- erated tokens for a variety of tasks. Evaluation results on publicly available machine transla- tion and response generation datasets demon- strate the effectiveness of our proposal. 1 Introduction Sequence-to-sequence (seq2seq) models are a dom- inant paradigm in various natural language genera- tion tasks, such as machine translation (Luong et al., 2015b; Tu et al., 2016), text summarization (Kiy- ono et al., 2018; Li et al., 2017), and response gen- eration (Miller et al., 2017; Pasunuru and Bansal, 2018). As Mi et al. (2016) reported, however, ba- sic seq2seq models (Bahdanau et al., 2015; Luong et al., 2015b) sometimes suffer from a repetition problem. One reason is that the attention mecha- nism does not explicitly consider which source side tokens have already been covered in the past atten- tions. As a result, the encoder repeatedly attends to the same token in the decoding steps, which leads to redundant generation. Many researchers have proposed variants of the seq2seq model to tackle the problem. The cover- age mechanism (Tu et al., 2016; Mi et al., 2016) prevented the model from generating redundant outputs by taking into account the coverage of the attention distribution. These approaches can be easily incorporated into the seq2seq model with only a single attention distribution between the en- coder and the decoder. However, for seq2seq mod- els with multiple attentions, such as Transformer (Vaswani et al., 2017), we cannot calculate the cov- erage of attentions, because the encoder attempts to attend to multiple attentions on each layer in the decoder. Thus, it is challenging to incorporate the coverage mechanism into the multi-attention based seq2seq models. As another solution, Suzuki and Nagata (2017) proposed word-frequency estimation (WFE) that predicts the upper-bound frequency for each output token from the given input tokens to control redun- dancy in the output. Furthermore, Kiyono et al. (2018) proposed a source-side prediction module (SPM) that estimates the occurrences of input to- kens from the hidden states of the decoder in the seq2seq model to reduce repetition. While WFE and SPM have an advantage in not depending on the structure of a seq2seq model, it is difficult to apply these approaches to some tasks other than text summarization because WFE and SPM assume that the input sentence contains more tokens than …

Content truncated to 500 words