Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2993–3001 June 6–11, 2021. ©2021 Association for Computational Linguistics 2993 Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management Zhengxu Hou1, Bang Liu2∗, Ruihui Zhao1, Zijing Ou3, Yafei Liu1, Xi Chen1, Yefeng Zheng1 1 Tencent Jarvis Lab 2 RALI & Mila, Université de Montréal 3Sun Yat-sen University holmes.hzx@gmail.com bang.liu@umontreal.ca ouzj@mail2.sysu.edu.cn {zacharyzhao,davenliu,jasonxchen,yefengzheng}@tencent.com Abstract For task-oriented dialog systems, training a Reinforcement Learning (RL) based Dialog Management module suffers from low sample efficiency and slow convergence speed due to the sparse rewards in RL. To solve this prob- lem, many strategies have been proposed to give proper rewards when training RL, but their rewards lack interpretability and cannot accurately estimate the distribution of state- action pairs in real dialogs. In this paper, we propose a multi-level reward modeling ap- proach that factorizes a reward into a three- level hierarchy: domain, act, and slot. Based on inverse adversarial reinforcement learning, our designed reward model can provide more accurate and explainable reward signals for state-action pairs. Extensive evaluations show that our approach can be applied to a wide range of reinforcement learning-based dialog systems and significantly improves both the performance and the speed of convergence. 1 Introduction Task-oriented dialog systems have become a fo- cal point in both academic and industrial research and have been playing a key role in conversa- tional assistants such as Amazon Alexa and Ap- ple’s Siri. (Budzianowski et al., 2018; Wei et al., 2018; Chen et al., 2019b) Existing research on task- oriented dialog systems mainly includes pipeline and end-to-end methods (Zhang et al., 2020). For pipeline-based systems, usually could be divided into four components: Natural Language Under- standing (NLU), Dialog State Tracking (DST), Dia- log Management (DM), and Natural Language Gen- eration (NLG). The modular structure makes the systems more interpretable and stable than end-to- end systems, which directly take natural language context as input and output a response. In pipeline-based dialog systems, DM is a core component that is responsible for modeling the cur- ∗Corresponding author. I would like to leave on Wednesday from Stevenage. I show a train leaving Stevenage new street at 17:40 and arriving at 20:23 on Wednesday. Will this work? Train Inform Leave 17:40 Train Inform Day Weds That will. Please make a booking for 5 people please. Okay, your reference number is A9NH Cool, I am also looking for restaurant with 4 stars and has free wifi. Train Inform Arrive 20:23 Train Inform Day Weds Train Inform People 5 Train Book Ref A9NH Hotel Inform Internet yes Hotel Inform Stars 4 I would like to leave on Wednesday from Stevenage. I find a train leaving Stevenage New Street at 17:40 and arriving at 20:23 on Wednesday. Will this work? That will. Please make a booking for 5 people please. Okay, your reference number is A9NH Cool, I am also looking for a hotel with 4 stars and free wifi. …

Content truncated to 500 words