IPA and STOUT: Leveraging Linguistic and Source-based Features for Machine Translation Evaluation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 394–401, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics IPA and STOUT: Leveraging Linguistic and Source-based Features for Machine Translation Evaluation Meritxell Gonz`alez, Alberto Barr´on-Cede˜no TALP Research Center, Technical University of Catalonia {mgonzalez,albarron}@lsi.upc.edu Llu´ıs M`arquez Qatar Computing Research Institute Qatar Foundation lmarquez@qf.org.qa Abstract This paper describes the UPC submissions to the WMT14 Metrics Shared Task: UPC- IPA and UPC-STOUT. These metrics use a collection of evaluation measures in- tegrated in ASIYA, a toolkit for machine translation evaluation. In addition to some standard metrics, the two submissions take advantage of novel metrics that consider linguistic structures, lexical relationships, and semantics to compare both source and reference translation against the candidate translation. The new metrics are available for several target languages other than En- glish. In the the official WMT14 evalua- tion, UPC-IPA and UPC-STOUT scored above the average in 7 out of 9 language pairs at the system level and 8 out of 9 at the segment level. 1 Introduction Evaluating Machine Translation (MT) quality is a difficult task, in which even human experts may fail to achieve a high degree of agreement when assessing translations. Conducting manual evalu- ations is impractical during the development cy- cle of MT systems or for transation applications addressed to general users, such as online transla- tion portals. Automatic evaluation measures bring valuable benefits in such situations. Compared to manual evaluation, automatic measures are cheap, more objective, and reusable across different test sets and domains. Nonetheless, automatic metrics are far from perfection: when used in isolation, they tend to stress specific aspects of the translation quality and neglect others (particularly during tuning); they are often unable to capture little system improve- ments (enhancements in very specific aspects of the translation process); and they may make un- fair comparisons when they are not able to reflect real differences among the quality of different MT systems (Gim´enez, 2008). ASIYA, the core of our approach, is an open- source suite for automatic machine translation evaluation and output analysis.1 It provides a rich set of heterogeneous metrics and tools to evalu- ate and analyse the quality of automatic transla- tions. The ASIYA core toolkit was first released in 2009 (Gim´enez and M`arquez, 2010a) and has been continuously improved and extended since then (Gonz`alez et al., 2012; Gonz`alez et al., 2013). In this paper we first describe the most recent enhancements to ASIYA: (i) linguistic-based met- rics for French and German; (ii) an extended set of source-based metrics for English, Spanish, Ger- man, French, Russian, and Czech; and (iii) the in- tegration of mechanisms to exploit the alignments between sources and translations. These enhance- ments are all available in ASIYA since version 3.0. We have used them to prepare the UPC submis- sions to the WMT14 Metrics Task: UPC-IPA and UPC-STOUT, which serve the purpose of testing their usefulness in a real comparative setting. The rest of the paper is structured as follows. Section 2 describes the …

Content truncated to 500 words