Machine Translation Evaluation Inside QARLA

Machine Translation Evaluation Inside QARLA Jes´us Gim´enez � , Enrique Amig´o � and Chiori Hori � � TALP Research Center, LSI Department Universitat Polit`ecnica de Catalunya � Departamento de Lenguajes y Sistemas Inform´aticos Universidad Nacional de Educaci´on a Distancia � InterACT Language Technologies Institute Carnegie Mellon University jgimenez@lsi.upc.edu, enrique@lsi.uned.es, chiori@cs.cmu.edu Abstract In this work we present the fundamentals of the IQMT frame- work for MT evaluation. IQMT offers a common work- bench on which existing evaluation metrics can be utilized. We suggest the IQ measure and test it on the Chinese-to- English data from the IWSLT 2004 Evaluation Campaign. We show how the correlation with human assessments at the system level improves substantially for most individual met- rics. Moreover, IQMT allows to robustly combine several met- rics avoiding scaling problems and metric weightings. Sev- eral metric combinations were tried, but correlations did not further improve significantly. 1. Introduction At the current level of improvement in a couple of years there will probably exist Machine Translation (MT) systems that perform better than humans according to existing MT eval- uation metrics. By then, these metrics, as they are currently applied, will become useless and more sophisticated metrics will be needed (Franz Och, talk at the ACL 2005 Workshop on “Building and Using Parallel Texts: Data-Driven Machine Translation and Beyond”). We refer to this problem as the ‘2008 MT Evaluation Challenge’. In this work we present the fundamentals of IQMT1 (In- side QARLA MT evaluation), a framework for MT evalua- tion which intends to overcome the MT evaluation challenge by offering a common workbench on which existing evalua- tion metrics can be used and combined. Inside QARLA [1], automatic evaluation of translations is interpreted as the application of similarity metrics between a set of candidate translations and a set of reference transla- tions. In this context, one of the main issues is to determine how similar a machine-produced translation must be to a set of human references to certify that it is a good translation. 1The IQMT package is publically available, released un- der the GNU Lesser General Public License (LGPL) of the Free Software Foundation, and may be freely downloaded at http://www.lsi.upc.edu/˜nlp/IQMT. That is, how the scale properties of the similarity metrics must be interpreted. Another important issue is how to combine information from different metrics into a single measure of quality. In the last years, it has been repeatedly argued that current MT eval- uation metrics do not capture well possible improvements at- tained by means of incorporating linguistic knowledge [2]. One of the possible reasons for that is that most of the cur- rent metrics are based on rewarding lexical similarity, thus not taking into account any additional syntactic or semantic information. We believe that new metrics should be inves- tigated and combined with current ones. The question then would be how to ponderate new similarity metrics with re- spect to existing ones. The QARLA framework has been successfully applied to the automatic evaluation of summaries [3]. Its probabilistic model …

Content truncated to 500 words