Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5998–6003 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5998 Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation Shun-Po Chuang1, Tzu-Wei Sung2, Alexander H. Liu1, and Hung-yi Lee1 1National Taiwan University, Taiwan 2University of California San Diego, USA {f04942141, b03902042, r07922013, hungyilee}@ntu.edu.tw Abstract Speech translation (ST) aims to learn transfor- mations from speech in the source language to the text in the target language. Previous works show that multitask learning improves the ST performance, in which the recognition decoder generates the text of the source language, and the translation decoder obtains the final trans- lations based on the output of the recognition decoder. Because whether the output of the recognition decoder has the correct semantics is more critical than its accuracy, we propose to improve the multitask ST model by utilizing word embedding as the intermediate. 1 Introduction Speech translation (ST) increasingly receives atten- tion from the machine translation (MT) commu- nity recently. To learn the transformation between speech in the source language and the text in the target language, conventional models pipeline au- tomatic speech recognition (ASR) and text-to-text MT model (B´erard et al., 2016). However, such pipeline systems suffer from error propagation. Previous works show that deep end-to-end mod- els can outperform conventional pipeline systems with sufficient training data (Weiss et al., 2017; In- aguma et al., 2019; Sperber et al., 2019). Neverthe- less, well-annotated bilingual data is expensive and hard to collect (Bansal et al., 2018a,b; Duong et al., 2016). Multitask learning plays an essential role in leveraging a large amount of monolingual data to improve representation in ST. Multitask ST models have two jointly learned decoding parts, namely the recognition and translation part. The recognition part firstly decodes the speech of source language into the text of source language, and then based on the output of the recognition part, the translation part generates the text in the target language. Vari- ant multitask models have been explored (Anas- tasopoulos and Chiang, 2018), which shows the improvement in low-resource scenario. Although applying the text of source language as the intermediate information in multitask end-to- end ST empirically yielded improvement, we argue whether this is the optimal solution. Even though the recognition part does not correctly transcribe the input speech into text, the final translation result would be correct if the output of the recognition part preserves sufficient semantic information for translation. Therefore, we explore to leverage word embedding as the intermediate level instead of text. In this paper, we apply pre-trained word embed- ding as the intermediate level in the multitask ST model. We propose to constrain the hidden states of the decoder of the recognition part to be close to the pre-trained word embedding. Prior works on word embedding regression show improved results on MT (Jauregi Unanue et al., 2019; Kumar and Tsvetkov, 2018). Experimental results show that the proposed approach obtains improvement to the …

Content truncated to 500 words