Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 38 - 45 July 10-15, 2022 ©2022 Association for Computational Linguistics Language Model Augmented Monotonic Attention for Simultaneous Translation Sathish Indurthi§∗ Mohd Abbas Zaidi‡ Beomseok Lee‡† Nikhil Kumar Lakumarapu‡† Sangha Kim‡ ‡Samsung Research, South Korea §Zoom AI Lab, Singapore sathishreddy.indurthi@zoom.us, {abbas.zaidi, bsgunn.lee, n07.kumar, sangha01.kim}@samsung.com Abstract The state-of-the-art adaptive policies for simul- taneous neural machine translation (SNMT) use monotonic attention to perform read/write decisions based on the partial source and target sequences. The lack of sufficient information might cause the monotonic attention to take poor read/write decisions, which in turn neg- atively affects the performance of the SNMT model. On the other hand, human translators make better read/write decisions since they can anticipate the immediate future words using linguistic information and domain knowledge. In this work, we propose a framework to aid monotonic attention with an external language model to improve its decisions. Experiments on MuST-C English-German and English-French speech-to-text translation tasks show the future information from language model improves the state-of-the-art monotonic multi-head attention model further. 1 Introduction A typical application of simultaneous neural ma- chine translation (SNMT) is conversational speech or live video caption translation. In order to achieve live translation, an SNMT model alternates be- tween performing read from source sequence and write to target sequence. For a model to decide whether to read or write at certain moment, either a fixed or an adaptive read/write policy can be used. Earlier approaches in simultaneous translation such as Ma et al. (2019a) and Dalvi et al. (2018) employ a fixed policy that alternate between read and write after the waiting period of k tokens. To alleviate possible long delay of fixed polices, re- cent works such as monotonic infinite lookback attention (MILk) (Arivazhagan et al., 2019), and monotonic multihead attention (MMA) (Ma et al., 2019c) developed flexible policies using monotonic attention (Raffel et al., 2017). ∗⋆Work done while at Samsung Research †Equal contribution Figure 1: The finetuned XLM-RoBERTa language model predicts German words using the prefix as in- put.(Green: Correct, Red: Incorrect, Black: Neutral). While these monotonic attention anticipates tar- get words using only available prefix source and target sequence, human translators anticipate the target words using their language expertise (linguis- tic anticipation) as well as contextual information (extra-linguistic anticipation) (Vandepitte, 2001). Inspired by human translation experts, we aim to augment monotonic attention with future informa- tion using language models (LM) (Devlin et al., 2019; Conneau et al., 2019). Integrating the external information effectively into text-to-text machine translation (MT) systems has been explored by several works (Khandelwal et al., 2020; Gulcehre et al., 2015, 2017; Stahlberg et al., 2018). Also, integrating future information implicitly into SNMT system during training is ex- plored in Wu et al. (2020) by simultaneously train- ing different wait-k SNMT systems. However, no previous works make use of explicit future informa- tion both during training and inference. To utilize explicit future information, we explored to …

Content truncated to 500 words