Learning to select data for transfer learning with Bayesian Optimization

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 372–382 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning to select data for transfer learning with Bayesian Optimization Sebastian Ruder♠♣ Barbara Plank♥ ♥Center for Language and Cognition, University of Groningen, The Netherlands ♠Insight Research Centre, National University of Ireland, Galway ♣Aylien Ltd., Dublin, Ireland sebastian@ruder.io,b.plank@rug.nl Abstract Domain similarity measures can be used to gauge adaptability and select suitable data for transfer learning, but existing ap- proaches define ad hoc measures that are deemed suitable for respective tasks. In- spired by work on curriculum learning, we propose to learn data selection measures using Bayesian Optimization and evaluate them across models, domains and tasks. Our learned measures outperform existing domain similarity measures significantly on three tasks: sentiment analysis, part- of-speech tagging, and parsing. We show the importance of complementing similar- ity with diversity, and that learned mea- sures are—to some degree—transferable across models, domains, and even tasks. 1 Introduction Natural Language Processing (NLP) models suf- fer considerably when applied in the wild. The distribution of the test data is typically very dif- ferent from the data used during training, caus- ing a model’s performance to deteriorate substan- tially. Domain adaptation is a prominent approach to transfer learning that can help to bridge this gap; many approaches were suggested so far (Blitzer et al., 2007; Daumé III, 2007; Jiang and Zhai, 2007; Ma et al., 2014; Schnabel and Schütze, 2014). However, most work focused on one-to- one scenarios. Only recently research consid- ered using multiple sources. Such studies are rare and typically rely on specific model transfer ap- proaches (Mansour, 2009; Wu and Huang, 2016). Inspired by work on curriculum learning (Ben- gio et al., 2009; Tsvetkov et al., 2016), we instead propose—to the best of our knowledge—the first model-agnostic data selection approach to trans- fer learning. Contrary to curriculum learning that aims at speeding up learning (see §6), we aim at learning to select the most relevant data from mul- tiple sources using data metrics. While several measures have been proposed in the past (Moore and Lewis, 2010; Axelrod et al., 2011; Van Asch and Daelemans, 2010; Plank and van Noord, 2011; Remus, 2012), prior work is limited in studying metrics mostly in isolation, using only the notion of similarity (Ben-David et al., 2007) and focus- ing on a single task (see §6). Our hypothesis is that different tasks or even different domains de- mand different notions of similarity. In this paper we go beyond prior work by i) studying a range of similarity metrics, including diversity; and ii) test- ing the robustness of the learned weights across models (e.g., whether a more complex model can be approximated with a simpler surrogate), do- mains and tasks (to delimit the transferability of the learned weights). The contributions of this work are threefold. First, we present the first model-independent ap- proach to learn a data selection measure for trans- fer learning. It outperforms baselines across three tasks and …

Content truncated to 500 words