Flow-Adapter Architecture for Unsupervised Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 92 - 102 May 22-27, 2022 c⃝2022 Association for Computational Linguistics [CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue Govardana Sachithanandam Ramachandran, Kazuma Hashimoto∗, Caiming Xiong Salesforce Research gramachandran@salesforce.com hassy@logos.t.u-tokyo.ac.jp cxiong@salesforce.com Abstract The recent success of reinforcement learning (RL) in solving complex tasks is often at- tributed to its capacity to explore and exploit an environment. Sample efficiency is usually not an issue for tasks with cheap simulators to sample data online. On the other hand, Task- oriented Dialogues (ToD) are usually learnt from offline data collected using human demon- strations. Collecting diverse demonstrations and annotating them is expensive. Unfortu- nately, RL policy trained on off-policy data are prone to issues of bias and generalization, which are further exacerbated by stochasticity in human response and non-markovian nature of annotated belief state of a dialogue man- agement system. To this end, we propose a batch-RL framework for ToD policy learn- ing: Causal-aware Safe Policy Improvement (CASPI). CASPI includes a mechanism to learn fine-grained reward that captures intention be- hind human response and also offers guaran- tee on dialogue policy’s performance against a baseline. We demonstrate the effectiveness of this framework on end-to-end dialogue task of the Multiwoz2.0 dataset. The proposed method outperforms the current state of the art. Fur- ther more we demonstrate sample efficiency, where our method trained only on 20% of the data, are comparable to current state of the art method trained on 100% data on two out of there evaluation metrics. 1 Introduction Offline task-oriented dialogue (ToD) systems in- volves solving disparate tasks of belief states track- ing, dialogue policy management, and response generation. Of these tasks, in this work we focus on dialogue policy management to improve the end- to-end performance of ToD. The need for sample ∗Contributed to this work during his time at Salesforce Research Code: https://github.com/salesforce/CASPI efficiency is key for learning offline task-oriented dialogue system, as access to data are finite and expensive. Recent advancements in off-policy rein- forcement learning methods that uses offline data as against a simulator has proven to be sample effi- cient (Thomas and Brunskill, 2016). The effective use of these techniques are hindered by the nature of ToD. For instance, bias correction in off-policy based methods usually requires estimation of be- haviour policy for a given state of Markov Deci- sion Process (MDP). In ToD, per-turn annotated belief-state does not capture the true state of the MDP. Example of such annotated belief-state are shown in Fig:1. Latent state information such as prosody, richness of natural language and among others induces stochasticity in the agents response. In addition to these short comings, the direct use of automatic evaluation metric as reward for pol- icy learning is not desirable, since these automatic evaluation metrics are often for the entire dialogue and not per turn. Hence such rewards are sparse and under-specified (Wang et al., 2020). Use of under-specified reward will often lead to …

Content truncated to 500 words