VERTa participation in the WMT14 Metrics Task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 368–375, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics VERTa participation in the WMT14 Metrics Task Elisabet Comelles Universitat de Barcelona Barcelona, Spain elicomelles@ub.edu Jordi Atserias Yahoo! Labs Barcelona, Spain jordi@yahoo-inc.com Abstract In this paper we present VERTa, a lin- guistically-motivated metric that com- bines linguistic features at different lev- els. We provide the linguistic motivation on which the metric is based, as well as describe the different modules in VERTa and how they are combined. Finally, we describe the two versions of VERTa, VERTa-EQ and VERTa-W, sent to WMT14 and report results obtained in the experiments conducted with the WMT12 and WMT13 data into English. 1 Introduction In the Machine Translation (MT) process, the evaluation of MT systems plays a key role both in their development and improvement. From the MT metrics that have been developed during the last decades, BLEU (Papineni et al., 2002) is one of the most well-known and widely used, since it is fast and easy to use. Nonetheless, researchers such as (Callison-Burch et al., 2006) and (Lavie and Dekowski, 2009) have claimed its weak- nesses regarding translation quality and its ten- dency to favour statistically-based MT systems. As a consequence, other more complex metrics that use linguistic information have been devel- oped. Some use linguistic information at lexical level, such as METEOR (Denkowski and Lavie, 2011); others rely on syntactic information, ei- ther using constituent (Liu and Hildea, 2005) or dependency analysis (Owczarzack et al., 2007a and 2007b; He et al., 2010); others use more complex information such as semantic roles (Giménez and Márquez, 2007 and 2008a; Lo et al., 2012). All these metrics focus on partial as- pects of language; however, other researchers have tried to combine information at different linguistic levels in order to follow a more holistic approach. Some of these metrics follow a ma- chine-learning approach (Leusch and Ney, 2009; Albrecht and Hwa, 2007a and 2007b), others combine a wide variety of metrics in a simple and straightforward way (Giménez, 2008b; Giménez and Márquez, 2010; Specia and Gimé- nez, 2010). However, very little research has been performed on the impact of the linguistic features used and how to combine this informa- tion from a linguistic point of view. Hence, our proposal is a linguistically-based metric, VERTa (Comelles et al., 2012), which uses a wide vari- ety of linguistic features at different levels, and aims at combining them in order to provide a wider and more accurate coverage than those metrics working at a specific linguistic level. In this paper we provide a description of the lin- guistic information used in VERTa, the different modules that form VERTa and how they are combined according to the language evaluated and the type of evaluation performed. Moreover, the two versions of VERTa participating in WMT14, VERTa-EQ and VERTa-W are de- scribed. Finally, for the sake of comparison, we use the data available in WMT12 and WMT13 to compare both versions …

Content truncated to 500 words