ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3721–3730 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3721 ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo and Xueqi Cheng CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences, Beijing, China {zhanghainan,lanyanyan, pangliang, guojiafeng, cxq}@ict.ac.cn Abstract In multi-turn dialogue generation, response is usually related with only a few contexts. Therefore, an ideal model should be able to detect these relevant contexts and produce a suitable response accordingly. However, the widely used hierarchical recurrent encoder- decoder models just treat all the contexts indis- criminately, which may hurt the following re- sponse generation process. Some researchers try to use the cosine similarity or the tradi- tional attention mechanism to find the rele- vant contexts, but they suffer from either insuf- ficient relevance assumption or position bias problem. In this paper, we propose a new model, named ReCoSa, to tackle this problem. Firstly, a word level LSTM encoder is con- ducted to obtain the initial representation of each context. Then, the self-attention mecha- nism is utilized to update both the context and masked response representation. Finally, the attention weights between each context and re- sponse representations are computed and used in the further decoding process. Experimen- tal results on both Chinese customer services dataset and English Ubuntu dialogue dataset show that ReCoSa significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on at- tention shows that the detected relevant con- texts by ReCoSa are highly coherent with hu- man’s understanding, validating the correct- ness and interpretability of ReCoSa. 1 Introduction This paper is concerned with the multi-turn dia- logue generation task, which is critical in many natural language processing (NLP) applications, such as customer services, intelligent assistant and chatbot. Recently, the hierarchical recurrent encoder-decoder (HRED) models (Serban et al., 2016; Sordoni et al., 2015) have been widely used in this area. In the encoding phase of these HRED The fitst example context1 你好，在吗？(Hello) context2 有什么问题我可以帮您呢?（What can I do for you?） context3 保真吗？（Is this product fidelity?） response 我们的商品都是海外采购的 绝对保证是正品的 (Our products are all purchased overseas Absolutely guaranteed to be genuine) The fitst example context1 我有个交易纠纷，麻烦你看看有进度吗 (I have a trading dispute. Could you please tell me whether it is progressing? ) context2 您好，请问是这个订单吗?(Hello, is this order?) context3 对(Yes) response 等待纠纷处理(Waiting for dispute resolution) Table 1: The two examples from the customer services dataset, and the red sentence indicates the relevant con- text to the response. models, a recurrent neural network (RNN) based encoder is first utilized to encode each input a con- text to a vector, and then a hierarchical RNN is conducted to encode these vectors to one vector. In the decoding phase, another RNN decoder is used to generate the response based on the above vec- tor. The parameters of both encoder and decoder …

Content truncated to 500 words