Conversational Word Embedding for Retrieval-Based Dialog System

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1375–1380 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1375 Conversational Word Embedding for Retrieval-Based Dialog System Wentao Ma†, Yiming Cui‡†, Ting Liu‡, Dong Wang†, Shijin Wang†§, Guoping Hu† †State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China ‡Research Center for Social Computing and Information Retrieval (SCIR), Harbin Institute of Technology, Harbin, China §iFLYTEK AI Research (Hebei), Langfang, China †§{wtma,ymcui,dongwang4,sjwang3,gphu}@iflytek.com ‡{ymcui,tliu}@ir.hit.edu.cn Abstract Human conversations contain many types of information, e.g., knowledge, common sense, and language habits. In this paper, we pro- pose a conversational word embedding method named PR-Embedding, which utilizes the con- versation pairs ⟨post, reply⟩ 1 to learn word embedding. Different from previous works, PR-Embedding uses the vectors from two dif- ferent semantic spaces to represent the words in post and reply. To catch the information among the pair, we first introduce the word alignment model from statistical machine translation to generate the cross-sentence win- dow, then train the embedding on word-level and sentence-level. We evaluate the method on single-turn and multi-turn response selection tasks for retrieval-based dialog systems. The experiment results show that PR-Embedding can improve the quality of the selected re- sponse. 2 1 Introduction Word embedding is one of the most fundamental work in the NLP tasks, where low-dimensional word representations are learned from unlabeled corpora. The pre-trained embeddings can reflect the semantic and syntactic information of words and help various downstream tasks get better per- formance (Collobert et al., 2011; Kim, 2014). The traditional word embedding methods train the models based on the co-occurrence statistics, such as Word2vec (Mikolov et al., 2013a,b), GloVe (Pennington et al., 2014). Those methods are widely used in dialog systems, not only in retrieval- based methods (Wang et al., 2015; Yan et al., 2016) but also the generation-based models (Serban et al., 1In this paper, we name the first utterance in the conversa- tion pair as ‘post,’ and the latter is ‘reply’ 2PR-Embedding source code is available at https:// github.com/wtma/PR-Embedding . 2016; Zhang et al., 2018b). The retrieval-based methods predict the answer based on the similarity of context and candidate responses, which can be divided into single-turn models (Wang et al., 2015) and multi-turn models (Wu et al., 2017; Zhou et al., 2018; Ma et al., 2019) based on the number of turns in context. Those methods construct the represen- tations of the context and response with a single vector space. Consequently, the models tend to select the response with the same words . On the other hand, as those static embeddings can not cope with the phenomenon of polysemy, researchers pay more attention to contextual rep- resentations recently. ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and XLNet (Yang et al., 2019) have achieved great success in many NLP tasks. However, it is difficult to apply them in the industrial dialog system due to their low computa- tional efficiency. In this paper, we focus on the static embedding, for it is …

Content truncated to 500 words