Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
20682Authors
nan
Abstract
AbstractComplex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However,cascaded models are known to be prone to error propagation and model discrepancy problems. Furthermore, there is no possibility of using end-to-end training data in conventional cascaded systems, meaning that the training data most suited for the task cannot be used. Previous studies suggested several approaches for integrated end-to-end training to overcome those problems, however they mostly rely on(synthetic or natural) three-way data. We propose a cascaded model based on the non-autoregressive Transformer that enables end-to-end training without the need for an explicit intermediate representation. This new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models and (ii) utilizes the end-to-end training data directly. We conduct an evaluation on two pivot-based machine translation tasks, namely French→German and German→Czech. Our experimental results show that the proposed architecture yields an improvement of more than 2 BLEU for French→German over the cascaded baseline.
Keywords
Full Content
Proceedings of the 18th International Conference on Spoken Language Translation, pages 39–45 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 39 NAIST English-to-Japanese Simultaneous Translation System for IWSLT 2021 Simultaneous Text-to-text Task Ryo Fukuda, Yui Oka, Yausumasa Kano, Yuki Yano, Yuka Ko, Hirotaka Tokuyama, Kosuke Doi, Sakriani Sakti, Katsuhito Sudoh, Satoshi Nakamura Nara Institute of Science and Technology, Nara, Japan fukuda.ryo.fo3@is.naist.jp Abstract This paper describes NAIST’s system for the English-to-Japanese Simultaneous Text-to- text Translation Task in IWSLT 2021 Eval- uation Campaign. Our primary submission is based on wait-k neural machine translation with sequence-level knowledge distillation to encourage literal translation. 1 Introduction Automatic simultaneous translation is an attractive research field that aims to translate an input before observing its end for real-time translation similar to human simultaneous interpretation. Starting from early attempts using rule-based machine translation (Matsubara and Inagaki, 1997; Ryu et al., 2006) and statistical methods using statistical machine translation (Bangalore et al., 2012; Fujita et al., 2013), recent studies successfully applied neural machine translation (NMT) into this task (Gu et al., 2017; Ma et al., 2019; Arivazhagan et al., 2019). The simultaneous translation shared task in the IWSLT evaluation campaign started on 2020 with English-to-German (Ansari et al., 2020) speech- to-text and text-to-text tasks, and a new language pair of English-to-Japanese has been included on 2021 only in text-to-text task. English-to-Japanese is much more challenging than English-to-German due to the large language difference in addition to data scarsity. We developed an automatic text-to-text simulta- neous translation system for this shared task. We applied some extensions to a standard wait-k NMT in the training time: sequence-level knowledge dis- tillation and target-side chunk shuffling. However, these techniques showed mixed results in different latency regimes on the IWSLT21 development set, so we configured the system differently for each latency regime. This paper describes the details of the system and the results on the development sets. We also describe our another attempt to include incremental constitutent label prediction that was not included in the primary system. 2 Simultaneous Neural Machine Translation with wait-k Let X = x1, x2, . . . , x|X| be an input sequence in a source language and Y = y1, y2, . . . , y|Y | be an output sequence in a target language. Here, the in- put can be speech or text, but we assume the input is text because this paper discusses the text-to-text task. The task of simultaneous translation is to translate X to Y incrementally. In other words, each output prediction of Y is made upon partial input observations of X. Suppose an output pre- fix subsequence Y j 1 = y1, y2, ..., yj has already been predicted from prefix observations of the in- put Xi 1 = x1, x2, ..., xi. When we predict the next output subsequence Y j ′ j+1 = yj+1, ..., yj′ after fur- ther partial observations Xi ′ i+1 = xi+1, ..., xi′, the prediction is made based on the following formula: Y …
Content truncated to 500 words
Labeled Text
AbstractComplex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However,cascaded models are known to be prone to error propagation and model discrepancy problems. Furthermore, there is no possibility of using end-to-end training data in conventional cascaded systems, meaning that the training data most suited for the task cannot be used. Previous studies suggested several approaches for integrated end-to-end training to overcome those problems, however they mostly rely on(synthetic or natural) three-way data. We propose a cascaded model based on the non-autoregressive Transformer that enables end-to-end training without the need for an explicit intermediate representation. This new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models and (ii) utilizes the end-to-end training data directly. We conduct an evaluation on two pivot-based machine translation tasks, namely French→German and German→Czech. Our experimental results show that the proposed architecture yields an improvement of more than 2 BLEU for French→German over the cascaded baseline.
Publication Details
- Paper ID:
- 20682
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan