Word Alignment by Fine-tuning Embeddings on Parallel Corpora

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2112–2128 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2112 Word Alignment by Fine-tuning Embeddings on Parallel Corpora Zi-Yi Dou, Graham Neubig Language Technologies Institute, Carnegie Mellon University {zdou,gneubig}@cs.cmu.edu Abstract Word alignment over parallel corpora has a wide variety of applications, including learn- ing translation lexicons, cross-lingual trans- fer of language processing tools, and auto- matic evaluation or analysis of translation out- puts. The great majority of past work on word alignment has worked by performing unsupervised learning on parallel text. Re- cently, however, other work has demonstrated that pre-trained contextualized word embed- dings derived from multilingually trained lan- guage models (LMs) prove an attractive alter- native, achieving competitive results on the word alignment task even in the absence of explicit training on parallel data. In this pa- per, we examine methods to marry the two ap- proaches: leveraging pre-trained LMs but fine- tuning them on parallel text with objectives designed to improve alignment quality, and proposing methods to effectively extract align- ments from these fine-tuned models. We per- form experiments on five language pairs and demonstrate that our model can consistently outperform previous state-of-the-art models of all varieties. In addition, we demonstrate that we are able to train multilingual word align- ers that can obtain robust performance on dif- ferent language pairs. Our aligner, AWE- SOME (Aligning Word Embedding Spaces Of Multilingual Encoders), with pre-trained models is available at https://github. com/neulab/awesome-align. 1 Introduction Word alignment is a useful tool to tackle a variety of natural language processing (NLP) tasks, including learning translation lexicons (Ammar et al., 2016; Cao et al., 2019), cross-lingual transfer of language processing tools (Yarowsky et al., 2001; Pad´o and Lapata, 2009; Tiedemann, 2014; Agi´c et al., 2016; Mayhew et al., 2017; Nicolai and Yarowsky, 2019), semantic parsing (Herzig and Berant, 2018) and The necessary correct ##ion will be made . Before Fine-tuning Eine entsprechende Änderung wird vor ##genommen werden . 0.89 0.82 0.78 0.80 0.77 0.77 0.78 0.79 0.83 0.85 0.82 0.81 0.78 0.77 0.79 0.79 0.78 0.80 0.84 0.84 0.77 0.74 0.78 0.80 0.82 0.80 0.78 0.80 0.88 0.86 0.81 0.81 0.78 0.78 0.79 0.78 0.81 0.82 0.82 0.77 0.76 0.77 0.76 0.77 0.78 0.81 0.84 0.77 0.76 0.73 0.74 0.75 0.77 0.80 0.80 0.78 0.79 0.75 0.76 0.75 0.76 0.73 0.77 0.95 The necessary correct ##ion will be made . After Fine-tuning 0.88 0.77 0.75 0.77 0.76 0.77 0.76 0.76 0.77 0.83 0.77 0.76 0.75 0.72 0.75 0.75 0.73 0.71 0.83 0.85 0.74 0.68 0.73 0.75 0.79 0.74 0.72 0.74 0.90 0.87 0.77 0.76 0.73 0.75 0.73 0.73 0.77 0.79 0.86 0.73 0.73 0.75 0.72 0.74 0.76 0.80 0.88 0.73 0.77 0.72 0.71 0.73 0.81 0.88 0.79 0.75 0.76 0.72 0.75 0.75 0.75 0.71 0.73 0.97 Figure 1: Cosine similarities between subword repre- sentations in a parallel sentence pair before and after fine-tuning. Red boxes indicate the gold alignments. speech recognition (Xu et al., 2019). …

Content truncated to 500 words