chrF: character n-gram F-score for automatic MT evaluation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. CHRF: character n-gram F-score for automatic MT evaluation Maja Popovi´c Humboldt University of Berlin Germany maja.popovic@hu-berlin.de Abstract We propose the use of character n-gram F-score for automatic evaluation of ma- chine translation output. Character n- grams have already been used as a part of more complex metrics, but their individual potential has not been investigated yet. We report system-level correlations with hu- man rankings for 6-gram F1-score (CHRF) on the WMT12, WMT13 and WMT14 data as well as segment-level correlation for 6- gram F1 (CHRF) and F3-scores (CHRF3) on WMT14 data for all available target lan- guages. The results are very promising, especially for the CHRF3 score – for trans- lation from English, this variant showed the highest segment-level correlations out- performing even the best metrics on the WMT14 shared evaluation task. 1 Introduction Recent investigations have shown that character level n-grams play an important role for auto- matic evaluation as a part of more complex met- rics such as MTERATER (Parton et al., 2011) and BEER (Stanojevi´c and Sima’an, 2014a; Stanojevi´c and Sima’an, 2014b). However, they have not been investigated as an individual metric so far. On the other hand, the n-gram based F-scores, especially the linguistically motivated ones based on Part-of-Speech tags and morphemes (Popovi´c, 2011), are shown to correlate very well with hu- man judgments clearly outperforming the widely used metrics such as BLEU and TER. In this work, we propose the use of the F- score based on character n-grams, i.e. the CHRF score. We believe that this score has a poten- tial as a stand-alone metric because it is shown to be an important part of the previously mentioned complex measures, and because, similarly to the morpheme-based F-score, it takes into account some morpho-syntactic phenomena. Apart from that, in contrast to the related metrics, it is sim- ple, it does not require any additional tools and/or knowledge sources, it is absolutely language inde- pendent and also tokenisation independent. The CHRF scores were calculated for all available translation outputs from the WMT12 (Callison-Burch et al., 2012), WMT13 (Bojar et al., 2013) and WMT14 (Bojar et al., 2014) shared tasks, and then compared with human rankings. System-level correlation coef- ficients are calculated for all data, segment-level correlations only for WMT14 data. The scores were calculated for all available target languages, namely English, Spanish, French, German, Czech, Russian and Hindi. 2 CHRF score The general formula for the CHRF score is: CHRFβ = (1 + β2) CHRP · CHRR β2 · CHRP + CHRR (1) where CHRP and CHRR stand for character n- gram precision and recall arithmetically averaged over all n-grams: • CHRP percentage of n-grams in the hypothesis which have a counterpart in the reference; • CHRR percentage of character n-grams in the refer- ence which are also present in the hypothesis. and β is a parameter which assigns β times more importance to recall …

Content truncated to 500 words