Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3472–3483, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3472 Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks Yufan Zhao1, Can Xu1∗, Wei Wu2 1Microsoft Corporation, Beijing, China 2Meituan, Beijing, China {yufzhao,caxu}@microsoft.com wuwei19850318@gmail.com Abstract We study multi-turn response generation for open-domain dialogues. The existing state-of- the-art addresses the problem with deep neural architectures. While these models improved response quality, their complexity also hin- ders the application of the models in real sys- tems. In this work, we pursue a model that has a simple structure yet can effectively lever- age conversation contexts for response gener- ation. To this end, we propose four auxil- iary tasks including word order recovery, ut- terance order recovery, masked word recovery, and masked utterance recovery, and optimize the objectives of these tasks together with max- imizing the likelihood of generation. By this means, the auxiliary tasks that relate to con- text understanding can guide the learning of the generation model to achieve a better local optimum. Empirical studies with three bench- marks indicate that our model can significantly outperform state-of-the-art generation models in terms of response quality on both automatic evaluation and human judgment, and at the same time enjoys a much faster decoding pro- cess. 1 Introduction As an important topic in conversational AI, open- domain human-machine conversation is gaining increasing attention from both academia and in- dustry. A common approach to building such a system is to learn a response generation model within an encoder-decoder framework using neu- ral sequence architectures (Sutskever et al., 2014; Vaswani et al., 2017). While the encoder-decoder framework has been successfully applied in vari- ous text generation tasks such as machine transla- tion (Vaswani et al., 2017), summarization (Rush et al., 2015), paraphrase generation (Dong et al., 2017), etc., it has to deal with a unique challenge ∗Corresponding author: Can Xu (caxu@microsoft.com). in the task of response generation: modeling con- versation contexts. A conversation context often exhibits a hierarchical structure with dependency existing on both a word-level and an utterance- level. Moreover, as indicated in (Xing et al., 2018; Zhang et al., 2019), information in a context is rather redundant for responding: commonly only a few words and utterances are useful for response generation, and the positions of the relevant words and utterances vary from case to case. To model the hierarchy of conversation contexts, hierarchical recurrent encoder-decoder (HRED) (Serban et al., 2016) extends the vanilla sequence-to-sequence model by a word-level encoder and an utterance- level encoder. Later on, a hierarchical recurrent attention network (HRAN) (Xing et al., 2018) har- nesses the decoder of the HRED model with word- level attention and utterance-level attention to dy- namically highlight the effect of relevant words and utterances in response synthesis. Very recently, ReCoSa (Zhang et al., 2019) further exploits multi- layer multi-head self-attention1 to model long-term dependency among utterances and responses. From HRED to HRAN, and then to ReCoSa, the perfor- mance …

Content truncated to 500 words