Soft Representation Learning for Sparse Transfer

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1560–1568 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1560 Soft Representation Learning for Sparse Transfer Haeju Park1 Jinyoung Yeo2 Gengyu Wang1 Seung-won Hwang1 Department of Computer Science, Yonsei University, Seoul, Korea1 T-Brain, AI Center, SK Telecom, Seoul, Korea2 {phj0225, posuer, seungwonh}@yonsei.ac.kr bradyeo33@sktbrain.com Abstract Transfer learning is effective for improving the performance of tasks that are related, and Multi-task learning (MTL) and Cross-lingual learning (CLL) are important instances. This paper argues that hard-parameter sharing, of hard-coding layers shared across different tasks or languages, cannot generalize well, when sharing with a loosely related task. Such case, which we call sparse transfer, might ac- tually hurt performance, a phenomenon known as negative transfer. Our contribution is us- ing adversarial training across tasks, to “soft- code” shared and private spaces, to avoid the shared space gets too sparse. In CLL, our pro- posed architecture considers another challenge of dealing with low-quality input. 1 Introduction Transfer learning in neural networks has been ap- plied in recent years to improving the performance of related tasks, for example, 1) multi-task learn- ing (MTL) with different tasks (labeled data avail- able all tasks) and 2) cross-lingual learning (CLL) with different language (but the same task) though labeled data available only in source language. For both settings, one of their most common strate- gies is hard-parameter sharing, as shown in Fig- ure 1a, which shares the hidden layers across tasks, which we will call shared layer. This ap- proach works well when tasks are closely related, when most features are invariant across tasks. Oth- erwise, which we call sparse transfer, transfer- ring between loosely related tasks often hurt per- formance, known as negative transfer. We elabo- rate this problem in MTL and CLL scenarios. First, for MTL, the shared space is reported to be sparse, in an architecture with one shared en- coder (Sachan and Neubig, 2018), when shared by K (e.g., K > 2 tasks) loosely-related tasks. To ad- dress this problem, as shown in Figure 1b, recent models (Liu et al., 2017; Lin et al., 2018) divide the features of different tasks into task-invariant and task-dependent latent spaces, which we will call shared and private spaces from this point on. However, since such approach still hard-codes shared and private features, deciding which sub- sets of tasks should share encoders in many-task settings, among all possible combinations of tasks, is a non-trivial design problem (Ruder et al., 2019; Sanh et al., 2019). Second, for CLL, the given task in source lan- guage (with rich resources) transfers to that for target languages without training resources. For the latter, machine-translated resources are fed in- stead, to the shared encoder (Schwenk and Douze, 2017; Conneau et al., 2018). When translation is perfect, the shared space would be dense: For ex- ample, English training pair with entailment rela- tionship, “Because it looked so formidable” and “It really did look wonderful” can be translated …

Content truncated to 500 words