LAYERED: Metric for Machine Translation Evaluation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 387–393, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics LAYERED: Metric for Machine Translation Evaluation Shubham Gautam Computer Science & Engineering, IIT Bombay shubhamg@cse.iitb.ac.in Pushpak Bhattacharyya Computer Science & Engineering, IIT Bombay pb@cse.iitb.ac.in Abstract This paper describes the LAYERED met- ric which is used for the shared WMT’14 metrics task. Various metrics exist for MT evaluation: BLEU (Papineni, 2002), ME- TEOR (Alon Lavie, 2007), TER (Snover, 2006) etc., but are found inadequate in quite a few language settings like, for ex- ample, in case of free word order lan- guages. In this paper, we propose an MT evaluation scheme that is based on the NLP layers: lexical, syntactic and seman- tic. We contend that higher layer met- rics are after all needed. Results are pre- sented on the corpora of ACL-WMT, 2013 and 2014. We end with a metric which is composed of weighted metrics at individ- ual layers, which correlates very well with human judgment. 1 Introduction Evaluation is an integral component of machine translation (MT). Human evaluation is difficult and time consuming so there is a need for a metric which can give the better evaluation in correlation to human judgement. There are several existing metrics such as: BLEU, METEOR etc. but these only deal with the lexical layer combining bag of words and n-gram based approach. We present an analysis of BLEU and the higher layer metrics on the ACL WMT 2013 corpora with 3 language pairs: French-English, Spanish- English and German-English. For syntactic layer, we considered three metrics: Hamming score, Kendall’s Tau distance score and the spearman rank score. Syntactic layer metrics take care of reordering within the words of the sentences so these may play an important role when there is a decision to be made between two MT output sentences of two different systems when both the sentences have same number of n-gram matches wrt the reference sentence but there is a differ- ence in the ordering of the sentence. We will dis- cuss these metrics in detail in the following sec- tions. The next NLP layer in consideration is the semantic layer which deals with the meaning of the sentences. For semantic layer, we considered two metrics: Shallow semantic score and Deep se- mantic score. On semantic layer, we considered entailment based measures to get the score. Ananthkrishnan et al. (2007) mentioned some issues in automatic evaluation using BLEU. There are some disadvantages of the existing metrics also such as: BLEU does not take care of reorder- ing of the words in the sentence. BLEU-like met- rics can give same score by permuting word or- der. These metrics can be unreliable at the level of individual sentences because there can be small number of n-grams involved. We would see in this paper that the correlation of BLEU is lower com- pared to the semantic layer metrics. Section 2 presents the study of related work in MT evaluation. Section 3 presents …

Content truncated to 500 words