Reinforced Training Data Selection for Domain Adaptation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1957–1968 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1957 Reinforced Training Data Selection for Domain Adaptation Miaofeng Liu♣∗†, Yan Song♠†, Hongbin Zou♦∗, and Tong Zhang♥ ♣MILA & DIRO, Universit´e de Montr´eal water3er@gmail.com ♠Tencent AI Lab clksong@gmail.com ♦School of Electronic Engineering, Xidian University hongbinzou@gmail.com ♥The Hong Kong University of Science and Technology tongzhang@ust.hk Abstract Supervised models suffer from the problem of domain shifting where distribution mismatch in the data across domains greatly affect model performance. To solve the problem, train- ing data selection (TDS) has been proven to be a prospective solution for domain adapta- tion in leveraging appropriate data. However, conventional TDS methods normally requires a predefined threshold which is neither easy to set nor can be applied across tasks, and models are trained separately with the TDS process. To make TDS self-adapted to data and task, and to combine it with model train- ing, in this paper, we propose a reinforcement learning (RL) framework that synchronously searches for training instances relevant to the target domain and learns better representations for them. A selection distribution generator (SDG) is designed to perform the selection and is updated according to the rewards computed from the selected data, where a predictor is included in the framework to ensure a task- specific model can be trained on the selected data and provides feedback to rewards. Experi- mental results from part-of-speech tagging, de- pendency parsing, and sentiment analysis, as well as ablation studies, illustrate that the pro- posed framework is not only effective in data selection and representation, but also general- ized to accommodate different NLP tasks. 1 Introduction Learning with massive data suffers from “Pyrrhic victory” where huge amounts of resource, e.g., computation, annotation, storage, etc., are con- sumed with many issues, one of which is that data quality considerably affects the performance of learned models. Especially in natural language ∗ This work was done during the internship of Miaofeng Liu and Hongbin Zou at Tencent AI Lab. † Corresponding authors. processing (NLP), such phenomenon is incredibly significant where noise and inaccurate annotations are demolishing models’ robustness when applying them across domains (Bollegala et al., 2011; Plank and Van Noord, 2011; Song and Xia, 2013; Ruder and Plank, 2018; Liu et al., 2018). Statistically, distribution mismatch is often observed between training and test data in such case. As a straightfor- ward solution to reduce the impact of the mismatch, TDS is effective for learning across domains (Ruder and Plank, 2017) by preventing negative transfer from irrelevant samples and noisy labels (Rosen- stein et al., 2005) while achieving equivalent perfor- mance with less computational efforts (Fan et al., 2017; Feng et al., 2018), especially when compared with learning-intensive domain adaptation methods such as sample reweighing (Borgwardt et al., 2006), feature distribution matching (Tzeng et al., 2014) and representation learning (Csurka, 2017). Although various TDS-based domain adapta- tion approaches were proposed for NLP tasks (Daum´e III, 2007; …

Content truncated to 500 words