SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1627 SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings Masoud Jalili Sabet∗1, Philipp Dufter∗1, Franc¸ois Yvon2, Hinrich Sch¨utze1 1 Center for Information and Language Processing (CIS), LMU Munich, Germany 2 Universit´e Paris-Saclay, CNRS, LIMSI, France {masoud,philipp}@cis.lmu.de,francois.yvon@limsi.fr Abstract Word alignments are useful for tasks like sta- tistical and neural machine translation (NMT) and cross-lingual annotation projection. Statis- tical word aligners perform well, as do meth- ods that extract alignments jointly with trans- lations in NMT. However, most approaches require parallel training data, and quality de- creases as less training data is available. We propose word alignment methods that require no parallel data. The key idea is to lever- age multilingual word embeddings – both static and contextualized – for word alignment. Our multilingual embeddings are created from monolingual data only without relying on any parallel data or dictionaries. We find that align- ments created from embeddings are superior for four and comparable for two language pairs compared to those produced by traditional sta- tistical aligners – even with abundant parallel data; e.g., contextualized embeddings achieve a word alignment F1 for English-German that is 5 percentage points higher than eflomal, a high-quality statistical aligner, trained on 100k parallel sentences. 1 Introduction Word alignments are essential for statistical ma- chine translation and useful in NMT, e.g., for im- posing priors on attention matrices (Liu et al., 2016; Chen et al., 2016; Alkhouli and Ney, 2017; Alkhouli et al., 2018) or for decoding (Alkhouli et al., 2016; Press and Smith, 2018). Further, word alignments have been successfully used in a range of tasks such as typological analysis (Lewis and Xia, 2008; ¨Ostling, 2015b), annotation projection (Yarowsky et al., 2001; Pad´o and Lapata, 2009; Asgari and Sch¨utze, 2017; Huck et al., 2019) and creating multilingual embeddings (Guo et al., 2016; Ammar et al., 2016; Dufter et al., 2018). ∗ Equal contribution - random order. Der Pinguin Nils Olav wurde vom norwegischen König zum Ritter geschlagen Pingvin Nils Olav Norvegiya qiroli tomonidan ritsar edi Sir Nils Olav III. です ペンギン knighted by el rey noruego Nils Olav der Dritte is a penguin nominato cavaliere par un roi norvégien Figure 1: Our method does not rely on parallel train- ing data and can align distant language pairs (German- Uzbek, top) and even mixed sentences (bottom). Exam- ple sentence is manually created. Algorithm: Itermax. Statistical word aligners such as the IBM mod- els (Brown et al., 1993) and their implementations Giza++ (Och and Ney, 2003), fast-align (Dyer et al., 2013), as well as newer models such as eflo- mal ( ¨Ostling and Tiedemann, 2016) are widely used for alignment. With the rise of NMT (Bahdanau et al., 2014), attempts have been made to interpret attention matrices as soft word alignments (Cohn et al., 2016; Koehn and Knowles, 2017; Ghader and Monz, 2017). Several methods create align- ments from attention matrices (Peter …

Content truncated to 500 words