Transfer Learning Between Related Tasks Using Expected Label Proportions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 31–42, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 31 Transfer Learning Between Related Tasks Using Expected Label Proportions Matan Ben Noach∗† and Yoav Goldberg∗‡ ∗Computer Science Department, Bar-Ilan University, Ramat-Gan Israel †Intel AI Lab, Petah-Tikva Israel ‡Allen Institute for Artificial Intelligence matan.ben.noach@intel.com, yoav.goldberg@gmail.com Abstract Deep learning systems thrive on abundance of labeled training data but such data is not al- ways available, calling for alternative methods of supervision. One such method is expecta- tion regularization (XR) (Mann and McCal- lum, 2007), where models are trained based on expected label proportions. We propose a novel application of the XR framework for transfer learning between related tasks, where knowing the labels of task A provides an es- timation of the label proportion of task B. We then use a model trained for A to label a large corpus, and use this corpus with an XR loss to train a model for task B. To make the XR framework applicable to large-scale deep-learning setups, we propose a stochastic batched approximation procedure. We demon- strate the approach on the task of Aspect- based Sentiment classification, where we ef- fectively use a sentence-level sentiment pre- dictor to train accurate aspect-based predictor. The method improves upon fully supervised neural system trained on aspect-level data, and is also cumulative with LM-based pretrain- ing, as we demonstrate by improving a BERT- based Aspect-based Sentiment model. 1 Introduction Data annotation is a key bottleneck in many data driven algorithms. Specifically, deep learning models, which became a prominent tool in many data driven tasks in recent years, require large datasets to work well. However, many tasks re- quire manual annotations which are relatively hard to obtain at scale. An attractive alternative is lightly supervised learning (Schapire et al., 2002; Jin and Liu, 2005; Chang et al., 2007; Grac¸a et al., 2007; Quadrianto et al., 2009a; Mann and Mc- Callum, 2010a; Ganchev et al., 2010; Hope and Shahaf, 2016), in which the objective function is supplemented by a set of domain-specific soft- constraints over the model’s predictions on unla- beled data. For example, in label regularization (Mann and McCallum, 2007) the model is trained to fit the true label proportions of an unlabeled dataset. Label regularization is special case of ex- pectation regularization (XR) (Mann and McCal- lum, 2007), in which the model is trained to fit the conditional probabilities of labels given features. In this work we consider the case of correlated tasks, in the sense that knowing the labels for task A provides information on the expected la- bel composition of task B. We demonstrate the ap- proach using sentence-level and aspect-level senti- ment analysis, which we use as a running example: knowing that a sentence has positive sentiment label (task A), we can expect that most aspects within this sentence (task B) will also have pos- itive label. While this expectation …

Content truncated to 500 words