Semi-supervised Stochastic Multi-Domain Learning using Variational Inference

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1923–1934 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1923 Semi-supervised Stochastic Multi-Domain Learning using Variational Inference Yitong Li Timothy Baldwin School of Computing and Information Systems The University of Melbourne, Australia yitongl4@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au Trevor Cohn Abstract Supervised models of NLP rely on large col- lections of text which closely resemble the in- tended testing setting. Unfortunately matching text is often not available in sufficient quantity, and moreover, within any domain of text, data is often highly heterogenous. In this paper we propose a method to distill the important do- main signal as part of a multi-domain learning system, using a latent variable model in which parts of a neural model are stochastically gated based on the inferred domain. We compare the use of discrete versus continuous latent vari- ables, operating in a domain-supervised or a domain semi-supervised setting, where the do- main is known only for a subset of training in- puts. We show that our model leads to substan- tial performance improvements over compet- itive benchmark domain adaptation methods, including methods using adversarial learning. 1 Introduction Text corpora are often collated from several dif- ferent sources, such as news, literature, micro- blogs, and web crawls, raising the problem of learning NLP systems from heterogenous data, and how well such models transfer to testing set- tings. Learning from these corpora requires mod- els which can generalise to different domains, a problem known as transfer learning or domain adaptation (Blitzer et al., 2007; Daum´e III, 2007; Joshi et al., 2012; Kim et al., 2016). In most state- of-the-art frameworks, the model has full knowl- edge of the domain of instances in the training data, and the domain is treated as a discrete in- dicator variable. However, in reality, data is often messy, with domain labels not always available, or providing limited information about the style and genre of text. For example, web-crawled cor- pora are comprised of all manner of text, such as news, marketing, blogs, novels, and recipes, how- ever the type of each document is typically not ex- plicitly specified. Moreover, even corpora that are labelled with a specific domain might themselves be instances of a much more specific area, e.g., “news” articles will cover politics, sports, travel, opinion, etc. Modelling these types of data accu- rately requires knowledge of the specific domain of each instance, as well as the domain of each test instance, which is particularly problematic for test data from previously unseen domains. A simple strategy for domain learning is to jointly learn over all the data with a single model, where the model is not conditioned on domain, and directly maximises p(y|x), where x is the text input, and y the output (e.g. classification label). Improvements reported in multi-domain learning (Daum´e III, 2007; Kim et al., 2016) have often focused on learning twin representations (shared and private representations) for each instance. The private representation is modelled …

Content truncated to 500 words