Using Spoken Word Posterior Features in Neural Machine Translation

Using Spoken Word Posterior Features in Neural Machine Translation Kaho Osamura1, Takatomo Kano1, Sakriani Sakti1,2, Katsuhito Sudoh1,2, Satoshi Nakamura1,2 1Nara Institute of Science and Technology, Japan 2RIKEN, Center for Advanced Intelligence Project AIP, Japan {osamura.kaho.oe5, kano.takatomo.km0, ssakti, sudoh, s-nakamura}@is.naist.jp Abstract A spoken language translation (ST) system consists of at least two modules: an automatic speech recognition (ASR) system and a machine translation (MT) system. In most cases, an MT is only trained and optimized using error-free text data. If the ASR makes errors, the translation accuracy will be greatly reduced. Existing studies have shown that training MT systems with ASR parameters or word lattices can improve the translation quality. However, such an ex- tension requires a large change in standard MT systems, re- sulting in a complicated model that is hard to train. In this paper, a neural sequence-to-sequence ASR is used as fea- ture processing that is trained to produce word posterior fea- tures given spoken utterances. The resulting probabilistic features are used to train a neural MT (NMT) with only a slight modification. Experimental results reveal that the pro- posed method improved up to 5.8 BLEU scores with synthe- sized speech or 4.3 BLEU scores with the natural speech in comparison with a conventional cascaded-based ST system that translates from the 1-best ASR candidates. 1. Introduction Spoken language translation is one innovative technology that allows people to communicate by speaking in their na- tive languages. However, translating a spoken language, in other words, recognizing speech and then translating words into another language, is incredibly complex. A standard ap- proach in speech-to-text translation systems requires effort to construct automatic speech recognition (ASR) and machine translation (MT), both of which are trained and tuned inde- pendently. ASR systems, which aim for the perfect transcription of utterances, are trained and tuned by minimizing the word error rate (WER) [1]. MT outputs are optimized and auto- matically measured based on a wide variety of metrics. One of the standard methods is the BLEU metric. However, all the errors from the words in ASR outputs are treated uni- formly without considering their syntactic roles, which are often critical for MT. Many studies have investigated the ef- fectiveness of the WER metric of ASR on the whole speech translation pipeline [2, 3, 4] and verified that ASR errors that compose the WER metric do not contribute equally to the BLEU score of translation quality. Furthermore, most MT systems are only trained and op- timized using error-free text data. Despite the fact that ASR technologies and their recognition rates have continued to improve, the occurrence of speech recognition errors remains inevitable. This is because there are many ambiguities due to a wide variety of acoustic and linguistic patterns produced by different speakers with various speaking styles and back- ground noises. If the ASR engine makes mistakes, the trans- lation accuracy will be significantly reduced. Thus, ignor- ing the existence of ASR errors while constructing a speech translation system is practically impossible. Previous research on traditional …

Content truncated to 500 words