Improving Word Alignment of Rare Words with Word Embeddings

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3209–3215, Osaka, Japan, December 11-17 2016. Improving Word Alignment of Rare Words with Word Embeddings Masoud Jalili Sabet(1) Heshaam Faili(1) (1) School of Electrical and Computer Engineering, University of Tehran, Iran (2) Faculty of Information Technology, Monash University, Australia {jalili.masoud,hfaili}@ut.ac.ir gholamreza.haffari@monash.edu Gholamreza Haffari(2) Abstract We address the problem of inducing word alignment for language pairs by developing an unsu- pervised model with the capability of getting applied to other generative alignment models. We approach the task by: i) proposing a new alignment model based on the IBM alignment model 1 that uses vector representation of words, and ii) examining the use of similar source words to overcome the problem of rare source words and improving the alignments. We apply our method to English-French corpora and run the experiments with different sizes of sentence pairs. Our results show competitive performance against the baseline and in some cases improve the results up to 6.9% in terms of precision. 1 Introduction Statistical machine translation systems usually break the translation task into two or more subtasks and an important one is finding word alignments over a sentence-aligned bilingual corpus (Brown et al., 1990). One of the common approaches to find word alignment, uses a generative translation model that produces a sentence in one language given the sentence in another language. Most SMT systems use an implementation of the IBM alignment models (Brown et al., 1993). These models use expectation maximization (EM) algorithm in training and only require sentence-aligned bilingual texts. Inducing word alignment from bilingual texts requires large amount of sentence-aligned parallel data which is not available for most of the language pairs. It also causes more problems for rare words which are the key parts of the sentences, but at the same time, are less frequent and therefore have a more biased probability distribution (Moore, 2004). This paper deals with the alignment task for the rare words by proposing a new alignment model based on the low-dimensional representations of the words. We explore the effect of replacing words with their vector space embeddings in IBM Alignment Model 1. In this model, instead of using a conditional multinomial distribution (to generate a target word ti ∈ T given a source word si ∈ S), we use a conditional Gaussian distribution and generate a d-dimensional word embedding Vti ∈ Rd given si. We then propose a method to improve the alignments for rare words by using their similar words and updating their distributions. The advantages of this model are: i) It uses monolingual word embedding which has more available training data; ii) By using the extracted knowledge from monolingual data, it has better results in low- resource word alignment tasks; iii) The Gaussian model can be applied to all generative alignment models by replacing the conditional distribution, and thus getting even better results. 2 Related Works 2.1 Word Embeddings Recent works on word embedding show improvements in capturing semantic features of …

Content truncated to 500 words