Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3715 - 3725 December 7-11, 2022 ©2022 Association for Computational Linguistics Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment Siyu Lai1∗, Zhen Yang2 , Fandong Meng2, Yufeng Chen1†, Jinan Xu1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {siyulai,chenyf,jaxu}@bjtu.edu.cn, {zieenyang,fandongmeng,withtomzhou}@tencent.com Abstract Word alignment which aims to extract lexicon translation equivalents between source and tar- get sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual lan- guage models. However, we find that the ex- isting approaches capture few interactions be- tween the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions be- tween the input sentence pairs, in which the source and target sentences are encoded sepa- rately with the shared self-attention modules in the shallow layers, while cross-lingual inter- actions are explicitly constructed by the cross- attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.1 1 Introduction Word alignment which aims to extract the lexicon translation equivalents between the input source- target sentence pairs (Brown et al., 1993; Zenkel et al., 2019; Garg et al., 2019; Jalili Sabet et al., 2020), has been widely used in machine transla- tion (Och and Ney, 2000; Arthur et al., 2016; Yang et al., 2020, 2021), transfer text annotations (Fang ∗Work done when Siyu were interning at Pattern Recog- nition Center, WeChat AI, Tencent Inc, China. †Yufeng Chen is the corresponding author. 1The code is publicly available at: https://github.com/ lisasiyu/Cross-Align ! israel Figure 1: An example from Dou and Neubig (2021). There is a misalignment between “以” and “to” and “for”. Red boxes denote the gold alignments. and Cohn, 2016; Huck et al., 2019), typological analysis (Lewis and Xia, 2008), generating adver- sarial examples (Lai et al., 2022), etc. Statistical word aligners based on the IBM translation models (Brown et al., 1993), such as GIZA++ (Och and Ney, 2003) and FastAlign (Dyer et al., 2013), have remained popular over the past thirty years for their good performance. Recently, with the advancement of deep neural models, neural aligners have devel- oped rapidly and surpassed the statistical aligners on many language pairs. Typically, these neural ap- proaches can be divided into two branches: Neural Machine Translation (NMT) based aligners and Language Model (LM) based aligners. NMT based aligners (Garg et al., 2019; Zenkel et al., 2020; Chen et al., …

Content truncated to 500 words