The Contribution of Linguistic Features to Automatic Machine Translation Evaluation

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 306–314, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP The Contribution of Linguistic Features to Automatic Machine Translation Evaluation Enrique Amig´o1 Jes´us Gim´enez2 Julio Gonzalo 1 Felisa Verdejo1 1UNED, Madrid {enrique,julio,felisa}@lsi.uned.es 2UPC, Barcelona jgimenez@lsi.upc.edu Abstract A number of approaches to Automatic MT Evaluation based on deep linguistic knowledge have been suggested. How- ever, n-gram based metrics are still to- day the dominant approach. The main reason is that the advantages of employ- ing deeper linguistic information have not been clarified yet. In this work, we pro- pose a novel approach for meta-evaluation of MT evaluation metrics, since correla- tion cofficient against human judges do not reveal details about the advantages and disadvantages of particular metrics. We then use this approach to investigate the benefits of introducing linguistic features into evaluation metrics. Overall, our ex- periments show that (i) both lexical and linguistic metrics present complementary advantages and (ii) combining both kinds of metrics yields the most robust meta- evaluation performance. 1 Introduction Automatic evaluation methods based on similarity to human references have substantially accelerated the development cycle of many NLP tasks, such as Machine Translation, Automatic Summariza- tion, Sentence Compression and Language Gen- eration. These automatic evaluation metrics allow developers to optimize their systems without the need for expensive human assessments for each of their possible system configurations. However, estimating the system output quality according to its similarity to human references is not a trivial task. The main problem is that many NLP tasks are open/subjective; therefore, different humans may generate different outputs, all of them equally valid. Thus, language variability is an issue. In order to tackle language variability in the context of Machine Translation, a considerable ef- fort has also been made to include deeper linguis- tic information in automatic evaluation metrics, both syntactic and semantic (see Section 2 for de- tails). However, the most commonly used metrics are still based on n-gram matching. The reason is that the advantages of employing higher linguistic processing levels have not been clarified yet. The main goal of our work is to analyze to what extent deep linguistic features can contribute to the automatic evaluation of translation quality. For that purpose, we compare – using four different test beds – the performance of 16 n-gram based metrics, 48 linguistic metrics and one combined metric from the state of the art. Analyzing the reliability of evaluation met- rics requires meta-evaluation criteria. In this re- spect, we identify important drawbacks of the standard meta-evaluation methods based on cor- relation with human judgements. In order to overcome these drawbacks, we then introduce six novel meta-evaluation criteria which represent dif- ferent metric reliability dimensions. Our analysis indicates that: (i) both lexical and linguistic met- rics have complementary advantages and different drawbacks; (ii) combining both kinds of metrics is a more effective and robust evaluation method across all meta-evaluation criteria. In addition, we also perform a qualitative analy- …

Content truncated to 500 words