Learning Transferable Feature Representations Using Neural Networks

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4124–4134 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4124 Learning Transferable Feature Representations Using Neural Networks Himanshu S. Bhatt∗ AI Labs, American Express Bengaluru, India himanshu.s.bhatt@aexp.com Arun Rajkumar IIT Madras Chennai, India arunr@cse.iitm.ac.in Shourya Roy∗ AI Labs, American Express Bengaluru, India shourya.roy@aexp.com Sriranjani Ramakrishnan Conduent Labs Bengaluru, India sriranjani.r@conduent.com Abstract Learning representations such that the source and target distributions appear as similar as possible has benefited transfer learning tasks across several applications. Generally it re- quires labeled data from the source and only unlabeled data from the target to learn such representations. While these representations act like a bridge to transfer knowledge learned in the source to the target; they may lead to negative transfer when the source specific characteristics detract their ability to represent the target data. We present a novel neural network architecture to simultaneously learn a two-part representation which is based on the principle of segregating source specific representation from the common representa- tion. The first part captures the source specific characteristics while the second part captures the truly common representation. Our archi- tecture optimizes an objective function which acts adversarial for the source specific part if it contributes towards the cross-domain learn- ing. We empirically show that two parts of the representation, in different arrangements, out- performs existing learning algorithms on the source learning as well as cross-domain tasks on multiple datasets. 1 Introduction Unsupervised domain adaptation is a sub field of machine learning where one learns from annotated data in a source domain with the aim of perform- ing well on non-annotated data in a target domain. This attractive feature wherein the data, distribu- tions and tasks may vary across domains has led to the widespread use of domain adaptation algo- rithms in several real world applications. A typical domain adaptation algorithm is provided with an- notated source data and non-annotated target data ∗ Research done while working for Xerox Research Cen- tre India. Figure 1: Illustrates the fundamental idea of learn- ing transferable feature representations behind the pro- posed technique. and it learns a ‘common representation’ where the source and target data distributions look similar. In this common representation, a model trained on the source data is expected to perform well on the target data as well. While learning a common representation is use- ful for transferring knowledge from the source to the target domain, this may often lead to ‘nega- tive transfer’ if we do not account for the funda- mental question “what to transfer”. It is observed that each domain has specific features that are highly discriminating only within a domain and contribute negatively if transferred across domains in a brute force manner (Pan and Yang, 2010), as shown in Figure 1. Traditional domain adap- tation algorithms, being oblivious to such source specific characteristics, learn common representa- tions which suffer from transfer loss as the source specific characteristics restrict their transferability. Moreover, it …

Content truncated to 500 words