Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision

Proceedings of the 29th International Conference on Computational Linguistics, pages 636–647 October 12–17, 2022. 636 Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation Yihe Wang1∗, Yitong Li2,3†, Yasheng Wang2, Fei Mi2 Pingyi Zhou2, Xin Wang1, Jin Liu1‡, Xin Jiang2, Qun Liu2 1School of Computer Science, Wuhan University 2Noah’s Ark Lab, Huawei 3Huawei Technologies Ltd. {yihewang, jinliu, xinwang0920}@whu.edu.cn {liyitong3, wangyasheng, feimi2, zhoupingyi, Jiang.Xin, qun.liu}@huawei.com Abstract Real human conversation data are complicated, heterogeneous, and noisy, from which building open-domain dialogue systems remains a chal- lenging task. In fact, such dialogue data still contains a wealth of information and knowl- edge, however, they are not fully explored. In this paper, we show existing open-domain dialogue generation methods that memorize context-response paired data with autoregres- sive or encode-decode language models under- utilize the training data. Different from current approaches, using external knowledge, we ex- plore a retrieval-generation training framework that can take advantage of the heterogeneous and noisy training data by considering them as "evidence". In particular, we use BERTScore for retrieval, which gives better qualities of the evidence and generation. Experiments over publicly available datasets demonstrate that our method can help models generate better re- sponses, even such training data are usually impressed as low-quality data. Such perfor- mance gain is comparable with those improved by enlarging the training set, even better. We also found that the model performance has a positive correlation with the relevance of the retrieved evidence. Moreover, our method per- formed well on zero-shot experiments, which indicates that our method can be more robust to real-world data. 1 Introduction Open-domain dialogue is a long-standing problem in natural language processing and has aroused the widespread interest of researchers. Many ap- proaches have been studied, and recently, genera- tion models trained on large-scale data have gained more attention (Adiwardana et al., 2020; Roller et al., 2020; Xu et al., 2021; Madotto et al., 2021; Bao et al., 2019, 2020; Zhang et al., 2019b; Wang ∗Work done during internship at Noah’s Ark Lab, Huawei †Equal Contribution ‡Corresponding Author et al., 2020). Open-domain dialogue systems are born to deal with diverse domains, and naturally their training data, usually crawled from online resources such as Reddit and Twitter, are hetero- geneous and contain utterances with many various topics, more freedom of topic shifting, and vague responses (Kummerfeld et al., 2018). As a result, directly building generation models from such data will be inefficient and usually requires "knowledge" during the training. One common solution is to introduce external knowledge, usually, in a form of unstructured knowledge passages from Wikipedia (Dinan et al., 2018) or Internet articles (Komeili et al., 2021), and then, to build retrieval-augmented generation (RAG) methods to improve the response quality (Lewis et al., 2020; Izacard and Grave, 2020). How- ever, this assumes knowledge-intensive scenarios, which are not suitable for general open-domain or robust to noise. According to our preliminary study, in the Reddit dataset, 43% of the dialogues are merely chitchat and cannot match "knowledge". Moreover, …

Content truncated to 500 words