LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics

Proceedings of NAACL-HLT 2018, pages 2070–2080 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics LSDSCC: A Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics Zhen Xu1, Nan Jiang2, Bingquan Liu1, Wenge Rong2, Bowen Wu3, Baoxun Wang3, Zhuoran Wang3, and Xiaolong Wang1 1School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China 2School of Computer Science and Engineering, Beihang University, Beijing, China 3Tricorn (Beijing) Technology Co., Ltd, Beijing, China {zxu,liubq,wangxl}@insun.hit.edu.cn {nanjiang, w.rong}@buaa.edu.cn {wubowen, wangbaoxun, wangzhuoran}@trio.ai Abstract It has been proven that automatic conversa- tional agents can be built up using the End- to-End Neural Response Generation (NRG) framework, and such a data-driven methodol- ogy requires a large number of dialog pairs for model training and reasonable evaluation met- rics for testing. This paper proposes a Large Scale Domain-Specific Conversational Corpus (LSDSCC) composed of high-quality query- response pairs extracted from the domain- specific online forum, with thorough pre- processing and cleansing procedures. Also, a testing set, including multiple diverse re- sponses annotated for each query, is con- structed, and on this basis, the metrics for measuring the diversity of generated results are further presented. We evaluate the per- formances of neural dialog models with the widely applied diversity boosting strategies on the proposed dataset. The experimental results have shown that our proposed corpus can be taken as a new benchmark dataset for the NRG task, and the presented metrics are promising to guide the optimization of NRG models by quantifying the diversity of the generated re- sponses reasonably. 1 Introduction Conversational agents (a.k.a. Chat-bots) are ef- fective media to establish communications with human beings and have received much attention from academic and industrial experts in recent years (Serban et al., 2017). One essential fact pro- moting the research work on conversational agents is the explosive growth of human interaction data accumulated in the social network services, such as Twitter1 and Reddit2. So, it is possible to build Chat-bots based on data-driven approaches (Ser- ban and Pineau, 2015). 1https://twitter.com/ 2https://www.reddit.com/ Nevertheless, there still remains a great chal- lenge for building such conversational agents: at present, the automatic evaluation metrics of NRG models can hardly afford to measure the seman- tic relevance and diversity of generated results reasonably, and even the latter evaluation aspect has been paid little attention. The widely ac- cepted evaluating methods employed by the exist- ing NRG models can be categorized as: a) metrics inherited from Machine Translation, e.g., BLEU, Perplexity, etc. (Yao et al., 2015; Lowe et al., 2017; Wu et al., 2018); b) discrete scores mea- suring the quality of generated results by human labeling (Shang et al., 2015; Serban et al., 2016; Xu et al., 2017); and c) case study comparing the generated results of different NRG models (Shang et al., 2015; Wang et al., 2017). The disappointing situation is that these evaluating methods have not revealed tangible difference among NRG models, the reasons for which can be reflected by the ex- ample given in …

Content truncated to 500 words