MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 83 - 91 May 26-27, 2022 c⃝2022 Association for Computational Linguistics MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning Di Wu Peking University, China inbath@163.com Liang Ding The University of Sydney, Australia liangding.liam@gmail.com Shuo Yang iFlytek Research, China shuoyang7@iflytek.com Mingyang Li Independent Researcher, China liamlmy@163.com Abstract Word alignment is essential for the downstream cross-lingual language understanding and gen- eration tasks. Recently, the performance of the neural word alignment models (Garg et al., 2019; Ding et al., 2019; Zenkel et al., 2020) has exceeded that of statistical models. How- ever, they heavily rely on sophisticated trans- lation models. In this study, we propose a su- per lightweight unsupervised word alignment model named MirrorAlign, in which a bidirec- tional symmetric attention trained with a con- trastive learning objective is introduced, and an agreement loss is employed to bind the attention maps, such that the alignments fol- low mirror-like symmetry hypothesis. Exper- imental results on several public benchmarks demonstrate that our model achieves compet- itive, if not better, performance compared to the state of the art in word alignment while significantly reducing the training and decod- ing time on average. Further ablation analysis and case studies show the superiority of our proposed MirrorAlign. Notably, we recognize our model as a pioneer attempt to unify bilin- gual word embedding and word alignments. Encouragingly, our approach achieves 16.4× speedup against GIZA++, and 50× parameter compression compared with the Transformer- based alignment methods. We release our code to facilitate the community1. 1 Introduction Word alignment, aiming to find the word-level cor- respondence between a pair of parallel sentences, is a core component of the statistical machine trans- lation (Brown et al., 1993, SMT). It also has ben- efited several downstream tasks, e.g., computer- aided translation (Dagan et al., 1993), semantic role labeling (Kozhevnikov and Titov, 2013), cross- lingual dataset creation (Yarowsky et al., 2001), cross-lingual modeling (Ding et al., 2020a), and cross-lingual text generation (Zan et al., 2022). 1https://github.com/moore3930/MirrorAlign Figure 1: Two examples of word alignment. The upper and bottom cases are the Chinese and Japanese refer- ences, respectively. Recently, in the era of neural machine transla- tion (Bahdanau et al., 2015; Vaswani et al., 2017, NMT), the attention mechanism plays the role of the alignment model in translation system. Un- fortunately, Koehn and Knowles (2017) show that attention mechanism may in fact dramatically di- verge with word alignment. The works of Ghader and Monz (2017); Li et al. (2019) also confirm this finding. Although there are some studies attempt to miti- gate this problem, most of them are rely on a sophis- ticated translation architecture (Garg et al., 2019; Zenkel et al., 2020). These methods are trained with a translation objective, which computes the probability of each target token conditioned on source tokens and previous target tokens. This will bring tremendous parameters and noisy alignments. Most recent work avoids the noisy alignment of translation models but employed too …

Content truncated to 500 words