Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 660–670 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 660 Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation Xinting Huang,1 Jianzhong Qi,1 Yu Sun,2 Rui Zhang1∗ 1The University of Melbourne, 2Twitter Inc. {xintingh@student., jianzhong.qi@, rui.zhang@}unimelb.edu.au, ysun@twitter.com Abstract Dialogue policy optimization often obtains feedback until task completion in task- oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only pro- vided at the end of dialogues. To address this issue, reward learning has been introduced to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. This approach requires complete state-action an- notations of human-to-human dialogues (i.e., expert demonstrations), which is labor inten- sive. To overcome this limitation, we propose a novel reward learning approach for semi- supervised policy learning. The proposed ap- proach learns a dynamics model as the re- ward function which models dialogue progress (i.e., state-action sequences) based on expert demonstrations, either with or without annota- tions. The dynamics model computes rewards by predicting whether the dialogue progress is consistent with expert demonstrations. We further propose to learn action embeddings for a better generalization of the reward function. The proposed approach outperforms competi- tive policy learning baselines on MultiWOZ, a benchmark multi-domain dataset. 1 Introduction Task-oriented dialogue systems complete tasks for users, such as making a restaurant reservation or finding attractions to visit, in multi-turn dialogues (Gao et al., 2018; Sun et al., 2016, 2017). Dialogue policy is a critical component in both the conven- tional pipeline approach (Young et al., 2013) and recent end-to-end approaches (Zhao et al., 2019). It decides the next action that a dialogue system should take at each turn. Considering its nature of sequential decision making, dialogue policy is usu- ally learned via reinforcement learning (Su et al., ∗Rui Zhang is the corresponding author. Table 1: State Action Annotation and Utterance Example User Side Utterance I would like moderate price range please. Dialogue State annotation Restaurant: {food=modern european, price range=moderate} System Side Utterance I found de luca cucina and riverside brasserie. does either of them sound good for you? System action annotation restaurant-inform:{name=de luca cucina, name=riverside brasserie} 2015; Peng et al., 2018; Zhang et al., 2019). Specif- ically, dialogue policy is learned by maximizing accumulated rewards over interactions with an en- vironment (i.e., actual users or a user simulator). Handcrafted rewards are commonly used for policy learning in earlier work (Peng et al., 2018), which assigns a small negative penalty at each turn and a large positive/negative reward when the task is successful/failed. However, such reward setting does not provide sufficient supervision signals in each turn other than the last turn, which causes the sparse reward issues and may result in poorly learned policies (Takanobu et al., 2019). To address this problem, reward function learn- ing that relies on expert demonstrations has been introduced (Takanobu et al., 2019; Li et al., 2019b). Specifically, state-action sequences generated by an …

Content truncated to 500 words