Towards Boosting the Open-Domain Chatbot with Human Feedback

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4060–4078 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Boosting the Open-Domain Chatbot with Human Feedback Hua Lu∗ Siqi Bao∗† Huang He Fan Wang Hua Wu Haifeng Wang Baidu Inc., China {luhua05, baosiqi, hehuang, wang.fan, wu_hua, wanghaifeng}@baidu.com Abstract Many open-domain dialogue models pre- trained with social media comments can gen- erate coherent replies but have difficulties pro- ducing engaging responses. This phenomenon might mainly result from the deficiency of an- notated human-human conversations and the misalignment with human preference. In this paper, we propose a novel and efficient frame- work Diamante to boost the open-domain chat- bot, where two kinds of human feedback (in- cluding explicit demonstration and implicit preference) are collected and leveraged. By asking annotators to select or amend the model- generated candidate responses, Diamante ef- ficiently collects the human demonstrated re- sponses and constructs a Chinese chit-chat dataset. To enhance the alignment with hu- man preference, Diamante leverages the im- plicit preference in the data collection process and introduces the generation-evaluation joint training. Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the perfor- mance of pre-trained dialogue models. The overall engagingness of the previous state-of- the-art model has been improved remarkably by 50% in Chinese open-domain conversations. 1 Introduction In recent years, the self-supervised pre-training based on tremendous unlabeled data has brought great success for many natural language process- ing tasks (Brown et al., 2020; Chowdhery et al., 2022). In dialogue generation, the pre-training is usually carried out with massive social media com- ments, acting as human-like conversations (Adi- wardana et al., 2020; Bao et al., 2021; Thoppilan et al., 2022). Despite that these pre-trained dia- logue models are capable of generating coherent replies, they have difficulties producing engaging ∗Equal contribution. †Corresponding author. responses. The main reasons for this phenomenon might be twofold. Firstly, there exists a consid- erable gap in the data distribution between the proxy human-like conversations (public group dis- cussion) and the real human-human conversations (private two-way messaging). Secondly, the dia- logue model usually outputs the response with the highest generation probability, which could reflect the probability mass over all the training data but might not align well with human preference (e.g., some biased or unsafe statements). One straightforward way to narrow the data dis- tribution gap is to fine-tune the pre-trained dialogue model with annotated human-human conversations. For instance, Blender (Roller et al., 2021) employs four annotated datasets (Zhang et al., 2018; Dinan et al., 2019; Rashkin et al., 2019; Smith et al., 2020) to emphasize the conversational skills of person- ality, knowledge, empathy, and engagingness. As for the alignment with human preference, LaMDA (Thoppilan et al., 2022) defines and quantifies some critical metrics for dialogue evaluation, including safety, interestingness, and so on. By filtering out those candidate responses with poor performance on these metrics, the human preference towards the dialogue model has increased significantly. How- ever, compared with …

Content truncated to 500 words