CUNI Experiments for WMT17 Metrics Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 604–611 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics CUNI Experiments for WMT17 Metrics Task David Mareˇcek Ondˇrej Bojar Ondˇrej H¨ubsch Rudolf Rosa Duˇsan Variˇs Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic surname@ufal.mff.cuni.cz, except hubschondrej@gmail.com Abstract In this paper, we propose three different methods for automatic evaluation of the machine translation (MT) quality. Two of the metrics are trainable on direct- assessment scores and two of them use de- pendency structures. The trainable metric AutoDA, which uses deep-syntactic fea- tures, achieved better correlation with hu- mans compared e.g. to the chrF3 metric. 1 Introduction With the ongoing research of the machine transla- tion (MT) systems in the past the need for accu- rate automatic evaluation of the translation quality became unquestionable. Even though the human judgment of the MT system outputs still holds as the most reliable form of evaluation, the high cost of human evaluation together with the amount of time required for such evaluation makes human judgment unsuitable for large scale experiments where we need to evaluate many different system configurations in a relatively short timespan. An additional important limitation of human evalua- tion is that it cannot be exactly repeated. This led to development of various methods for automatic MT evaluation in the past with the aim to elimi- nate the need for the expensive human assessment of the developed MT systems. In this paper we suggest three novel methods for automatic MT evaluation together with their direct comparison: 1. AutoDA: A linear regression model using semantic features trained on WMT Direct Assessment scores (Bojar et al., 2016) or HUMEseg scores (Birch et al., 2016). 2. TreeAggreg: N-gram based metric computed over aligned syntactic structures instead of the linear representation of the translated sen- tences. 3. NMTScorer: A neural sequence classifier which assigns correct/incorrect flags to the evaluated sentence segments. Table 1 shows the main properties of the pro- posed methods. Some of them were mainly devel- oped for Czech as the target language and were later modified to be applied to other languages. The differences in the data preprocessing and their impact on the resulting evaluator are also de- scribed in this paper. 2 AutoDA: Automatic Direct Assessment AutoDA is a sentence-level metric trainable on any direct assessment scores. The metric is based on a simple linear regression combining several features extracted from the automatically aligned translation-reference pair. There may be also other established metrics within the features. The training data with golden direct-assessment scores available are shown in Table 2. We describe two variants. The first one works only on Czech and uses many semantic features based of rich Czech tectogrammatical annotation (B¨ohmov´a et al., 2003). The second one uses much fewer features, however, it is language uni- versal and needs only a dependency parsing model available. 2.1 AutoDA Using Czech Tectogrammatics This metric automatically …

Content truncated to 500 words