Machine Translation Evaluation for Arabic using Morphologically-enriched Embeddings

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1398–1408, Osaka, Japan, December 11-17 2016. Machine Translation Evaluation for Arabic using Morphologically-Enriched Embeddings Francisco Guzmán, Houda Bouamor∗, Ramy Baly† and Nizar Habash‡ Qatar Computing Research Institute ∗Carnegie Mellon University in Qatar †American University of Beirut ‡New York University Abu Dhabi fguzman@qf.org.qa, hbouamor@cmu.edu, rgb15@mail.aub.edu, nizar.habash@nyu.edu Abstract Evaluation of machine translation (MT) into morphologically rich languages (MRL) has not been well studied despite posing many challenges. In this paper, we explore the use of embeddings ob- tained from different levels of lexical and morpho-syntactic linguistic analysis and show that they improve MT evaluation into an MRL. Specifically we report on Arabic, a language with complex and rich morphology. Our results show that using a neural-network model with different input representations produces results that clearly outperform the state-of-the-art for MT evaluation into Arabic, by almost over 75% increase in correlation with human judgments on pairwise MT evaluation quality task. More importantly, we demonstrate the usefulness of morpho-syntactic representations to model sentence similarity for MT evaluation and address complex linguistic phenomena of Arabic. 1 Introduction Statistical machine translation (SMT) into morphologically rich languages (MRL) faces many chal- lenges: from handling a complex and rich vocabulary, to designing adequate MT metrics that take morphology into account. While the first problem has widely explored (e.g. by using morphological analysis tools to reduce sparsity), the evaluation part has only been partly addressed. This is problematic since traditional MT metrics struggle to distinguish between (i) incorrect lexical choices; (ii) valid alter- native lexical or syntactic variations; and (iii) differences in morphological inflection that are the result of incorrect case assignment or morphological agreement. While metrics like METEOR (Denkowski and Lavie, 2011) have made it possible to distinguish between (i) and (ii) by using paraphrases, (iii) is still an open problem. As a result, progress in SMT for MRL is hindered by the lack of adequate evaluation metrics. Since SMT metrics are used not only for evaluation but also for tuning system parameters, it is crucial that the MT metrics correctly handle morphology. Most recently, deep learning models have been used more heavily in different parts of the natural lan- guage processing (NLP) community, including MT and MT evaluation. One of the main advantages of such models is the use of distributed word representations (embeddings). It has been shown that word embeddings are able to capture to certain semantic and syntactic aspects of words (Mikolov et al., 2013). Further refinements allow the inclusion of morphological information into distributed representations (Cotterell and Schütze, 2015). Word embeddings have been shown to help with modeling textual simi- larity well in the context of MT evaluation for MT into English (Guzmán et al., 2015), and community Question Answering (Guzmán et al., 2016). Nonetheless little exploration has been done on the use of embeddings for MT into MRL. In this paper, we investigate how embeddings obtained from different levels of lexical and morpho- syntactic linguistic analysis can improve MT …

Content truncated to 500 words