Sharpness-Aware Minimization Improves Language Model Generalization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 680 - 694 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning When to Translate for Streaming Speech Qianqian Dong1∗, Yaoming Zhu1∗, Mingxuan Wang1, Lei Li2 † 1 ByteDance AI Lab 2 University of California, Santa Barbara {dongqianqian,zhuyaoming,wangmingxuan.89}@bytedance.com leili@cs.ucsb.edu Abstract How to find proper moments to generate par- tial sentence translation given a streaming speech input? Existing approaches waiting- and-translating for a fixed duration often break the acoustic units in speech, since the bound- aries between acoustic units in speech are not even. In this paper, we propose MoSST, a simple yet effective method for translat- ing streaming speech content. Given a usu- ally long speech sequence, we develop an efficient monotonic segmentation module in- side an encoder-decoder model to accumulate acoustic information incrementally and detect proper speech unit boundaries for the input in speech translation task. Experiments on multiple translation directions of the MuST-C dataset show that MoSST outperforms exist- ing methods and achieves the best trade-off be- tween translation quality (BLEU) and latency. Our code is available at https://github. com/dqqcasia/mosst. 1 Introduction Speech translation (ST) aims at translating from source language speech into target language text, which is widely helpful in various scenarios such as conference speeches, business meetings, cross- border customer service, and overseas travel. There are two kinds of application scenarios, including the non-streaming translation and the streaming one. The non-streaming models can listen to the complete utterances at one time and then generate the translation afterward. While, the streaming models need to balance the latency and quality and generate translations based on the partial utterance, as shown in Figure 1. Recently, end-to-end approaches have achieved remarkable progress in non-streaming ST. Previ- ous work (Weiss et al., 2017; Bérard et al., 2018; Livescu and Goldwater, 2019; Bansal et al., 2019; ∗Equal contribution. †Work is done while at ByteDance. I remember my first fire. Ich erinnere mich an mein erstes Feuer. Listen Write Figure 1: An illustration of streaming speech-to-text translation. ST models listen to the audio in source lan- guage, and generate tokens in target language. Alinejad and Sarkar, 2020; Stoian et al., 2020) Ansari et al. (2020) has shown that an end-to-end model achieves even better performance compared to the cascaded competitors. However, attempts at end-to-end streaming ST are still not fully explored. Traditional streaming ST is usually formed by cas- cading a streaming speech recognition module with a streaming machine translation module (Oda et al., 2014; Dalvi et al., 2018). Most of the previous work focuses on simultaneous text translation (Gu et al., 2017a). Ma et al. (2019a) propose a novel wait-k strategy based on the prefix-to-prefix frame- work, which is one of the popular research methods of simultaneous text translation. For end-to-end streaming ST, Ma et al. (2020b); Ren et al. (2020); Ma et al. (2021a) introduce the methodology of streaming machine translation into streaming ST and formalize the task, which belongs to the first study …

Content truncated to 500 words