Neural Speech Translation at AppTek

Neural Speech Translation at AppTek Evgeny Matusov, Patrick Wilken, Parnia Bahar∗, Julian Schamper∗, Pavel Golik, Albert Zeyer∗, Joan Albert Silvestre-Cerd`a+, Adri`a Mart´ınez-Villaronga+, Hendrik Pesch, and Jan-Thorsten Peter Applications Technology (AppTek), Aachen, Germany {ematusov,pwilken,pbahar,jschamper,pgolik,jsilvestre,amartinez,hpesch,jtpeter}@apptek.com ∗Also RWTH Aachen University, Germany +Also Universitat Polit`ecnica de Val`encia, Spain Abstract This work describes AppTek’s speech translation pipeline that includes strong state-of-the-art automatic speech recog- nition (ASR) and neural machine translation (NMT) com- ponents. We show how these components can be tightly coupled by encoding ASR confusion networks, as well as ASR-like noise adaptation, vocabulary normalization, and implicit punctuation prediction during translation. In another experimental setup, we propose a direct speech translation approach that can be scaled to translation tasks with large amounts of text-only parallel training data but a limited num- ber of hours of recorded and human-translated speech. 1. Introduction AppTek participated in the evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT) 2018 with the goal of obtaining best possible speech translation quality by streamlining the interface be- tween ASR and machine translation (MT). We tested a new way of encoding multiple hypotheses of ASR as input to an NMT system. We also experimented with a novel direct neu- ral translation model that translates source language speech into target language text, while at the same time benefiting from text-only parallel training data in a multi-task learning framework. To make these experiments possible, we made sure that our NMT system can handle different types of input, and its source language vocabulary is harmonized w.r.t. the ASR system vocabulary. We also fine-tuned the NMT model on ASR-like noise, making it more robust against recognition errors. Finally, we tested different punctuation prediction ap- proaches and found that the implicit prediction of punctua- tion marks by the MT component works best in our setting. Although improving our state-of-the-art NMT model was not our main focus, we benefited from fine-tuning the model on the in-domain data, as well as from ensembles of mod- els which differ in architecture – recurrent neural network (RNN) model with attention [1] or transformer architec- ture [2] – and/or input modality – ASR confusion network (CN) or first-best ASR output. This paper is organized as follows. We start by reviewing related work in 2, point- ing out some differences and novelties in our approach. In Section 3, we describe our methods for data filtering, pre- processing, and punctuation prediction. Section 4 gives an overview of our ASR system. Section 5 describes the details of AppTek’s NMT system. Section 6 gives details of how ASR confusion networks can be encoded as an input of the NMT system. In Section 7, we describe our direct speech translation prototype. The results of our speech translation experiments are summarized in Section 8. 2. Related Work Theoretical background for tighter coupling of statistical ASR and MT systems had been first published in [3]. In prac- tice, it was realized e. g. as statistical phrase-based transla- tion of ASR word lattices with acoustic and language model scores …

Content truncated to 500 words