Multi-Task Networks with Universe, Group, and Task Feature Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 820–830 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 820 Multi-Task Networks With Universe, Group, and Task Feature Learning Shiva Pentyala∗ Texas A&M University pk123@tamu.edu Mengwen Liu Amazon Alexa mengwliu@amazon.com Markus Dreyer Amazon Alexa mddreyer@amazon.com Abstract We present methods for multi-task learning that take advantage of natural groupings of re- lated tasks. Task groups may be defined along known properties of the tasks, such as task do- main or language. Such task groups represent supervised information at the inter-task level and can be encoded into the model. We inves- tigate two variants of neural network architec- tures that accomplish this, learning different feature spaces at the levels of individual tasks, task groups, as well as the universe of all tasks: (1) parallel architectures encode each input si- multaneously into feature spaces at different levels; (2) serial architectures encode each in- put successively into feature spaces at different levels in the task hierarchy. We demonstrate the methods on natural language understand- ing (NLU) tasks, where a grouping of tasks into different task domains leads to improved performance on ATIS, Snips, and a large in- house dataset. 1 Introduction In multi-task learning (Caruana, 1993), multiple related tasks are learned together. Rather than learning one task at a time, multi-task learning uses information sharing between multiple tasks. This technique has been shown to be effective in multiple different areas, e.g., vision (Zhang et al., 2014), medicine (Bickel et al., 2008), and natural language processing (Collobert and Weston, 2008; Luong et al., 2016; Fan et al., 2017). The selection of tasks to be trained together in multi-task learning can be seen as a form of super- vision: The modeler picks tasks that are known a priori to share some commonalities and decides to train them together. In this paper, we consider the case when information about the relationships of ∗ This work was done while Shiva Pentyala was interning at Amazon Alexa. these tasks is available as well, in the form of nat- ural groups of these tasks. Such task groups can be available in various multi-task learning scenar- ios: In multi-language modeling, when learning to parse or translate multiple languages jointly, infor- mation on language families would be available; in multimodal modeling, e.g., when learning text tasks and image tasks jointly, clustering the tasks into these two groups would be natural. In multi- domain modeling, which is the focus of this pa- per, different tasks naturally group into different domains. We hypothesize that adding such inter-task su- pervision can encourage a model to generalize along the desired task dimensions. We introduce neural network architectures that can encode task groups, in two variants: • Parallel network architectures encode each input simultaneously into feature spaces at different levels; • serial network architectures encode each in- put successively into feature spaces at differ- ent levels in the task hierarchy. These neural network architectures are general and can be …

Content truncated to 500 words