VERTa: a Linguistically-motivated Metric at the WMT15 Metrics Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 366–372, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. VERTa: a Linguistically-motivated Metric at the WMT15 Metrics Task Elisabet Comelles GRIAL Universitat de Barcelona (UB) Barcelona Spain elicomelles@ub.edu Jordi Atserias IXA Group University of the Basque Country (UPV/EHU) Spain jordi_atserias001@ehu.eus Abstract This paper describes VERTa’s submis- sion to the 2015 EMNLP Workshop on Statistical Machine Translation. VERTa is a linguistically-motivated metric that combines linguistic features at different levels. In this paper, VERTa is described briefly, as well as the three versions submitted to the workshop: VERTa- 70Adeq30Flu, VERTa-EQ and VERTa- W. Finally, the experiments conducted with the WMT14 data are reported and some conclusions are drawn. 1 Introduction In the last decade Automatic Machine Transla- tion (MT) Evaluation has become a key field in Natural Language Processing due to the amount of texts that are translated over the world and the need for a quick, reliable and inexpensive way to evaluate the quality of the output text. Therefore, a large number of metrics have been developed, which range from very simple metrics to more complex ones. Within simple metrics there are those that do not use any type of linguistic in- formation, such as BLEU (Papineni et al., 2002) which is one of the most well-known and widely used, since it is fast and easy to use. Other me- trics though, rely on linguistic information used at lexical level such as METEOR (Denkowski and Lavie, 2014); at syntactic level, using either constituent analysis (Liu and Hildea, 2005) or dependency analysis (Owczarzack et al., 2007a and 2007b; He et al., 2010); while others use more complex information such as semantic roles (Giménez and Márquez, 2007 and 2008; Lo et al., 2012). However, all these metrics focus on partial aspects of language which might lead to a biased evaluation. As a consequence, in the last years researchers have been exploring different ways to combine a wide variety of linguistic fea- tures, either using machine-learning techniques (Leusch and Ney, 2009; Albrecht and Hwa, 2007a and 2007b; Gautam and Bhattacharyya, 2014; Joty et al., 2014) or in a more simple and straightforward way (Giménez, 2008; Giménez and Márquez, 2010; Specia and Giménez, 2010, González et al., 2014). Nevertheless, little re- search has been carried out in order to explore the suitability of the linguistic features used and how they should be combined, from a linguistic point of view. Therefore, this paper proposes a new version of VERTa, a linguistically- motivated metric (Comelles and Atserias, 2014) which uses a wide variety of linguistic features at different levels and which aims at moving away from a biased evaluation and providing a more holistic approach to MT evaluation. Last year VERTa participated in the WMT15 and achieved promising results at system level, this year we would like to improve the metric’s performance at segment level. To this aim, a Language Model Module has been added, as well as a NERC component. In this paper we …

Content truncated to 500 words