A Human Judgement Corpus and a Metric for Arabic MT Evaluation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 207–213, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics A Human Judgment Corpus and a Metric for Arabic MT Evaluation Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit and Kemal Oflazer Carnegie Mellon University in Qatar {hbouamor,halshikh,behrang,ko}@cmu.edu Abstract We present a human judgments dataset and an adapted metric for evaluation of Arabic machine translation. Our medium- scale dataset is the first of its kind for Ara- bic with high annotation quality. We use the dataset to adapt the BLEU score for Arabic. Our score (AL-BLEU) provides partial credits for stem and morphologi- cal matchings of hypothesis and reference words. We evaluate BLEU, METEOR and AL-BLEU on our human judgments cor- pus and show that AL-BLEU has the high- est correlation with human judgments. We are releasing the dataset and software to the research community. 1 Introduction Evaluation of Machine Translation (MT) contin- ues to be a challenging research problem. There is an ongoing effort in finding simple and scal- able metrics with rich linguistic analysis. A wide range of metrics have been proposed and evaluated mostly for European target languages (Callison- Burch et al., 2011; Mach´aˇcek and Bojar, 2013). These metrics are usually evaluated based on their correlation with human judgments on a set of MT output. While there has been growing interest in building systems for translating into Arabic, the evaluation of Arabic MT is still an under-studied problem. Standard MT metrics such as BLEU (Pa- pineni et al., 2002) or TER (Snover et al., 2006) have been widely used for evaluating Arabic MT (El Kholy and Habash, 2012). These metrics use strict word and phrase matching between the MT output and reference translations. For morpholog- ically rich target languages such as Arabic, such criteria are too simplistic and inadequate. In this paper, we present: (a) the first human judgment dataset for Arabic MT (b) the Arabic Language BLEU (AL-BLEU), an extension of the BLEU score for Arabic MT evaluation. Our annotated dataset is composed of the output of six MT systems with texts from a diverse set of topics. A group of ten native Arabic speakers an- notated this corpus with high-levels of inter- and intra-annotator agreements. Our AL-BLEU met- ric uses a rich set of morphological, syntactic and lexical features to extend the evaluation beyond the exact matching. We conduct different exper- iments on the newly built dataset and demonstrate that AL-BLEU shows a stronger average correla- tion with human judgments than the BLEU and METEOR scores. Our dataset and our AL-BLEU metric provide useful testbeds for further research on Arabic MT and its evaluation.1 2 Related Work Several studies on MT evaluation have pointed out the inadequacy of the standard n-gram based eval- uation metrics for various languages (Callison- Burch et al., 2006). For morphologically complex languages and those without word delimiters, sev- eral studies have attempted to improve upon them and suggest more reliable metrics that correlate better with human judgments …

Content truncated to 500 words