Embedding-Enhanced GIZA++: Improving Low-Resource Word Alignment Using Embeddings

Embedding-Enhanced GIZA++: Improving Word Alignment Using Embeddings Kelly Marchisio kmarc@jhu.edu Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21211, USA Conghao Xiong∗ chxiong21@cse.cuhk.edu.hk Department of Computer Science and Engineering, The Chinese University of Hong Kong, New Territory, HKSAR Philipp Koehn phi@jhu.edu Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21211, USA Abstract A popular natural language processing task decades ago, word alignment has been domi- nated until recently by GIZA++, a statistical method based on the 30-year-old IBM mod- els. New methods that outperform GIZA++ primarily rely on large machine translation mod- els, massively multilingual language models, or supervision from GIZA++ alignments it- self. We introduce Embedding-Enhanced GIZA++, and outperform GIZA++ without any of the aforementioned factors. Taking advantage of monolingual embedding spaces of source and target language only, we exceed GIZA++’s performance in every tested scenario for three languages pairs. In the lowest-resource setting, we outperform GIZA++ by 8.5, 10.9, and 12 AER for Ro-En, De-En, and En-Fr, respectively. We release our code at https: //github.com/kellymarchisio/ee-giza. 1 Introduction Word alignment techniques were once ubiquitous in the machine translation (MT) literature, as they formed a critical part of statistical machine translation (SMT) systems. Since the advent of neural machine translation (NMT), word alignment is no longer a step in typical NMT training, but is still important for other tasks such as annotation transfer (e.g. Yarowsky and Ngai, 2001; Rasooli et al., 2018), as a post-processing step of MT to reinsert markup (e.g. M¨uller, 2017), and for some mapping-based unsupervised MT methods such as Artetxe et al. (2019). GIZA++ (Och, 2003), a statistical alignment model, has been the most commonly used tool for word alignment quality for 20 years and is based the IBM translation models that are yet a decade older (Brown et al., 1993). Though a handful of neural systems have outperformed GIZA++, these rely on large MT models (e.g. Stengel-Eskin et al., 2019; Chen et al., 2020; Zenkel et al., 2020), massively multilingual language models (e.g. Garg et al., 2019b; Jalili Sa- bet et al., 2020; Dou and Neubig, 2021), supervision from human-annotated alignments (Nagata et al., 2020), or combinations of the above. We introduce Embedding-Enhanced GIZA++ (EE-GIZA++), an improvement to GIZA++ without any of the aforementioned factors. EE-GIZA++ biases GIZA++ to align semantically similar words from a shared embedding space. We outperform GIZA++ in all tested settings ∗ Work completed at Johns Hopkins University. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 264 on three language pairs. EE-GIZA++ is particularly strong in comparison with GIZA++ when parallel training data is scarce: using only ∼500 lines of bitext, it outperforms GIZA++ by 10.9 AER1 and 12.0 AER for De-En and Fr-En, respectively. 2 Related Work Fast-align is a statistical aligner similar to GIZA++. It is a reparameterization of IBM Model 2 (Dyer et al., 2013). eflomal is another highly-performant non-neural aligner ( ¨Ostling and Tiede- mann, 2016). We use GIZA++ …

Content truncated to 500 words