Affective and Contextual Embedding for Sarcasm Detection

Proceedings of the 28th International Conference on Computational Linguistics, pages 15–25 Barcelona, Spain (Online), December 8-13, 2020 15 Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism Pan Xie∗1 Zhi Cui2 Xiuying Chen3 Xiaohui Hu4 Jianwei Cui2 Bin Wang2 1Beihang University 2Xiaomi AI Lab 3Peking University 4Chinese Academy of Sciences panxie@buaa.edu.cn xy-chen@pku.edu.cn hxh@iscas.ac.cn {cuizhi,cuijianwei,wangbin11}@xiaomi.com Abstract Non-autoregressive models generate target words in a parallel way, which achieve a faster de- coding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promising approach is to train a conditional masked translation model (CMTM), and refine the generated results within several iterations. Unfortunately, such approach hardly considers the sequential dependency among target words, which inevitably results in a translation degradation. Hence, instead of solely training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse sequential information into it. Concretely, we insert a left-to-right mask to the same decoder of CMTM, and then induce it to autoregressively review whether each generated word from CMTM is supposed to be replaced or kept. The experimental results (WMT14 En↔De and WMT16 En↔Ro) demonstrate that our model uses dramatically less training computations than the typical CMTM, as well as outperforms several state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge distillation, our model even surpasses a typical left-to-right Transformer model, while significantly speeding up decoding. 1 Introduction Neural Machine Translation (NMT) models have achieved a great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014; Kalchbrenner et al., 2016; Gehring et al., 2017; Vaswani et al., 2017). Typically, NMTs use autoregressive decoders, where the words are generated one-by-one. However, due to the left-to-right dependency, this computationally-intensive decoding process cannot be easily parallelized, and therefore causes a large latency (Gu et al., 2018). To break the bottleneck of autoregression, several non-autoregressive models have been proposed to induce the decoder to generate all target words simultaneously (Gu et al., 2018; Łukasz Kaiser et al., 2018; Li et al., 2019; Ma et al., 2019). Despite the acceleration of computation efficiency, these models usually suffers from the cost of translation accuracy. Even worse, they decode a target only in one shot, and thus miss a chance to remedy a flawed translation. Against them, a promising research line is to perform refinement on the generated result within several iterations (Lee et al., 2018; Ghazvininejad et al., 2019). Along this line, Ghazvininejad et al. (2019) propose a Mask-Predict decoding strategy, which iteratively refines the generated translation given the most confident target words predicted from the previous iteration. This model is trained using an objective of conditional masked translation modeling (CMTM), by predicting the masked words conditioned on the rest of observed words. However, CMTM just learns from a subset of words instead of the entire target in terms of a training step. As a result, it will iterate more times over the training dataset to explore the contextual relationship within a sentence, and thus will struggle with …

Content truncated to 500 words