Generative latent neural models for automatic word alignment

Generative latent neural models for automatic word alignment Anh Khoa Ngo Ho anh-khoa.ngo-ho@limsi.fr Franc¸ois Yvon francois.yvon@limsi.fr Universit´e Paris-Saclay, CNRS, LIMSI Bˆat. 508, rue John von Neumann, Campus Universitaire, F-91405 Orsay Abstract Word alignments identify translational correspondences between words in a parallel sentence pair and are used, for instance, to learn bilingual dictionaries, to train statistical machine trans- lation systems or to perform quality estimation. Variational autoencoders have been recently used in various of natural language processing to learn in an unsupervised way latent represen- tations that are useful for language generation tasks. In this paper, we study these models for the task of word alignment and propose and assess several evolutions of a vanilla variational autoencoders. We demonstrate that these techniques can yield competitive results as compared to Giza++ and to a strong neural network alignment system for two language pairs. 1 Introduction Word alignment is one of the basic tasks in multilingual Natural Language Processing (NLP) and is used to learn bilingual dictionaries, to train statistical machine translation (SMT) sys- tems (Koehn, 2010), to filter out noise from translation memories (Pham et al., 2018) or in quality estimation applications (Specia et al., 2017). Word alignments can also be viewed as a form of possible explanation of the often opaque behavior of a Neural Machine Translation (Stahlberg et al., 2018). Word alignment aims to identify translational equivalences at the level of individual lexical units (Och and Ney, 2003; Tiedemann, 2011) in parallel sentences. Successful alignment models either rely on bilingual association measures parameterizing a combinatorial problem (eg. an optimal matching in a bipartite graph); or on probabilistic models, as represented by the IBM Models of Brown et al. (1993) and the HMM model of Vogel et al. (1996). All these models use unsupervised learning to estimate the likelihood of alignment links at the word level from large collections of parallel sentences. Such approaches are typically challenged by low-frequency words, whose co-occurrences are poorly estimated; they also fail to take into account context information in alignment; finally, they make assumptions that are overly simplistic (eg. that all alignments are one-to-many or many-to-one), especially when the languages under focus belong to different linguistic families. Even though their overall performance seem fair for related languages (eg. French-English), there is still much room for improving. Indeed, the error rate of automatic alignments tools such as Giza++ (Och and Ney, 2003) or Fastalign (Dyer et al., 2013), even for high resource languages, is still well above 15-20%; and the situation is much worse in low-resource settings (Martin et al., 2005; Xiang et al., 2010; McCoy and Frank, 2018). As for most NLP applications (Collobert et al., 2011), and notably for machine translation (Cho et al., 2014; Bahdanau et al., 2015), neural-based approaches offer new opportunities to Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 64 reconsider some of these issues. Following up on the work of eg. (Yang …

Content truncated to 500 words