Tied Multitask Learning for Neural Speech Translation

Proceedings of NAACL-HLT 2018, pages 82–91 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Tied Multitask Learning for Neural Speech Translation Antonios Anastasopoulos and David Chiang Department of Computer Science and Engineeering University of Notre Dame {aanastas,dchiang}@nd.edu Abstract We explore multitask models for neural trans- lation of speech, augmenting them in order to reflect two intuitive notions. First, we in- troduce a model where the second task de- coder receives information from the decoder of the first task, since higher-level intermediate representations should provide useful infor- mation. Second, we apply regularization that encourages transitivity and invertibility. We show that the application of these notions on jointly trained models improves performance on the tasks of low-resource speech transcrip- tion and translation. It also leads to better per- formance when using attention information for word discovery over unsegmented input. 1 Introduction Recent efforts in endangered language documen- tation focus on collecting spoken language re- sources, accompanied by spoken translations in a high resource language to make the resource in- terpretable (Bird et al., 2014a). For example, the BULB project (Adda et al., 2016) used the LIG- Aikuma mobile app (Bird et al., 2014b; Blachon et al., 2016) to collect parallel speech corpora be- tween three Bantu languages and French. Since it’s common for speakers of endangered languages to speak one or more additional languages, collec- tion of such a resource is a realistic goal. Speech can be interpreted either by transcrip- tion in the original language or translation to an- other language. Since the size of the data is ex- tremely small, multitask models that jointly train a model for both tasks can take advantage of both signals. Our contribution lies in improv- ing the sequence-to-sequence multitask learning paradigm, by drawing on two intuitive notions: that higher-level representations are more useful than lower-level representations, and that transla- tion should be both transitive and invertible. Higher-level intermediate representations, such as transcriptions, should in principle carry infor- mation useful for an end task like speech transla- tion. A typical multitask setup (Weiss et al., 2017) shares information at the level of encoded frames, but intuitively, a human translating speech must work from a higher level of representation, at least at the level of phonemes if not syntax or semantics. Thus, we present a novel architecture for tied mul- titask learning with sequence-to-sequence models, in which the decoder of the second task receives information not only from the encoder, but also from the decoder of the first task. In addition, transitivity and invertibility are two properties that should hold when mapping be- tween levels of representation or across languages. We demonstrate how these two notions can be im- plemented through regularization of the attention matrices, and how they lead to further improved performance. We evaluate our models in three experiment settings: low-resource speech transcription and translation, word discovery on unsegmented in- put, and high-resource text translation. Our high- resource experiments are performed on English, French, and German. Our low-resource speech …

Content truncated to 500 words