Translation vs. Dialogue: A Comparative Analysis of Sequence-to-Sequence Modeling

Proceedings of the 28th International Conference on Computational Linguistics, pages 4111–4122 Barcelona, Spain (Online), December 8-13, 2020 4111 Translation vs. Dialogue: A Comparative Analysis of Sequence-to-Sequence Modeling Wenpeng Hu1,∗, Ran Le2,∗, Bing Liu2,†, Jinwen Ma1, Dongyan Zhao2, Rui Yan2,‡ 1 Department of Information Science, Peking University 2 Wangxuan Institute of Computer Technology, Peking University {wenpeng.hu, leran, dcsliub, jwma, zhaody, ruiyan}@pku.edu.cn Abstract Understanding neural models is a major topic of interest in the deep learning community. In this paper, we propose to interpret a general neural model comparatively. Specifically, we study the sequence-to-sequence (Seq2Seq) model in the contexts of two mainstream NLP tasks–machine translation and dialogue response generation–as they both use the seq2seq model. We investigate how the two tasks are different and how their task difference results in major differences in the behaviors of the resulting translation and dialogue generation systems. This study allows us to make several interesting observations and gain valuable insights, which can be used to help develop better translation and dialogue generation models. To our knowledge, no such comparative study has been done so far. 1 Introduction The sequence-to-sequence model (seq2seq), especially its enhanced variants of Long Short-Term Mem- ory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Bahdanau et al., 2014), have been demonstrated to be highly effective for a variety of NLP tasks, e.g., machine translation (MT) (Sutskever et al., 2014), dialogue response generation (DRG) (Li et al., 2016b), and many others. A seq2seq model takes in a source sentence (a sequence) and generates a target sentence (another se- quence). This model was first proposed for MT by translating a source language into a target language. It is now also widely used for DRG to “translate” an input utterance from the user to an output response. Traditional feature-engineering solutions to MT and DRG require a large number of handcrafted fea- tures. In end-to-end learning, seq2seq models require almost no manual features for MT or DRG. The model is purely driven by data: when trained on the translation data, the model learns to translate; when trained on the conversational data, the model learns to converse. However, this approach also makes the neural model hard to understand or interpret. To this end, researchers have proposed to understand and visualize recurrent neural networks (Karpathy et al., 2015) and neural machine translation (MT) models (Ding et al., 2017). As we know, MT and DRG are two very different tasks. But the same seq2seq modeling works for both, which is counter-intuitive. However, recent studies for understanding neural models have primarily focused on a single task. This paper proposes to study the seq2seq model comparatively in the contexts of two tasks, MT and DRG. Since the two very different tasks use the same model, the internal network behaviors of the model must be very different for the tasks. Our comparative study shows the contrast, which enables us to see a clearer picture of the model for each task and its issues, which cannot be easily observed from the …

Content truncated to 500 words