UHH Submission to the WMT17 Metrics Shared Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 582–588 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics UHH Submission to the WMT17 Metrics Shared Task Melania Duma and Wolfgang Menzel University of Hamburg Natural Language Systems Division {duma, menzel}@informatik.uni-hamburg.de Abstract In this paper the UHH submission to the WMT17 Metrics Shared Task is presented, which is based on sequence and tree ker- nel functions applied to the reference and candidate translations. In addition we also explore the effect of applying the ker- nel functions on the source sentence and a back-translation of the MT output, but also on the pair composed of the candi- date translation and a pseudo-reference of the source segment. The newly proposed metric was evaluated using the data from WMT16, with the results demonstrating a high correlation with human judgments. 1 Introduction The evaluation of Machine Translation (MT) rep- resents a very important domain of research, as providing meaningful, automatic and accurate methods for determining the quality of machine- translated output is a key component in the devel- opment cycle of a MT system. However, the task is inherently difficult due to the expressiveness of natural language, which often allows conveying a message in more than one equivalent ways. When translating from a source language into a target one, the input data for evaluation conventionally consists of a set of tuples, with each tuple com- posed of: • a source segment, representing the sentence to be translated in the source language • a candidate translation (also known as a target segment), obtained by translating the source segment into the target language using an MT system • a reference translation, representing a correct human-generated translation of the source segment As a research field, MT evaluation can be di- vided into two categories: reference-free evalu- ation and reference-based one. The reference- free evaluation, also known as Quality Estimation, aims at providing automatic methods, for assess- ing the quality of candidate translations, which do not require reference translations. In the case of a reference-based evaluation, the target seg- ment is compared with the reference translation resulting in a score that measures the similarity between the two sentences. Different approaches for computing the comparison have been imple- mented, with the most frequently used one be- ing BLEU (Papineni et al., 2002), which measures the quality of the candidate translation by count- ing the number of n-grams it has in common with the reference translations. Nonetheless, multiple disadvantages of BLEU have already been pointed out, as in Callison-Burch et al. (2006), where it is shown that an increase of the BLEU score does not necessary correlate with a better performing sys- tem. This has motivated further research into ad- ditional MT evaluation methods that rely on more than lexical matching by additionally including the syntactic and semantic structure of the sen- tences (e.g. (Popovi´c and Ney, 2009), (Gautam and Bhattacharyya, 2014) ). We propose a new method for the evaluation of MT output, …

Content truncated to 500 words