chrF++: words helping character n-grams

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 612–618 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics CHRF ++: words helping character n-grams Maja Popovi´c Humboldt University of Berlin, Germany maja.popovic@hu-berlin.de Abstract Character n-gram F-score (CHRF) shown to correlate very well with hu-is man relative rankings of different machine translation outputs, especially for morpho- logically rich target languages. However, its relation with direct human assessments is not yet clear. In this work, Pearson’s correlation coefficients for direct assess- ments are investigated for two currently available target languages, English and Russian. First, different β parameters (in range from 1 to 3) are re-investigated with direct assessment, and it is confirmed that β = 2 is the optimal option. Then sepa- rate character and word n-grams are inves- tigated, and the main finding is that, apart from character n-grams, word 1-grams and 2-grams also correlate rather well with direct assessments. Further experiments show that adding word unigrams and bi- grams to the standard CHRF score im- proves the correlations with direct assess- ments, though it is still not clear which option is better, unigrams only (CHRF+) or unigrams and bigrams (CHRF++). This should be investigated in future work on more target languages. 1 Introduction Recent investigations (Popovi´c, 2015; Stanojevi´c et al., 2015; Popovi´c, 2016; Bojar et al., 2016) have shown that the character n-gram F-score (CHRF) represents a very promising evaluation metric for machine translation, especially for mor- phologically rich target languages – it is fast, it does not require any additional tools or informa- tion, it is language independent and tokenisation independent, and it correlates very well with hu- man relative rankings (RR) (Callison-Burch et al., 2008). In order to produce these rankings, human annotators have to decide which sentence trans- lation is better/worse than another without giving any note about the absolute quality of any of the evaluated translations. This type of human judg- ment has been the offical evaluation metric and gold standard for all automatic metrics at WMT shared tasks from 2008 until 2016. Another type of human judgment, direct human assessment (DA) (Bojar et al., 2016), has become additional official evaluation metric for WMT-16, and the only one for WMT-17. These assessments consist of absolute quality scores for each trans- lated sentence. Contrary to RR, the relation be- tween CHRF and DA has still not been investigated systematically. Preliminary experiments in previ- ous work (Popovi´c, 2016) shown that, concern- ing DA, the main advantage of character-based F- score CHRF in comparison to word-based F-score WORDF is better correlation for good translations for which WORDF often assigns too low scores. In this work, we systematically investigate re- lations between DA and both character and word n-grams, as well as their combinations. The scores are calculated for all available translation outputs from the WMT-15 and WMT-16 shared tasks (Bojar et al., 2016) which contain two target languages, English (translated from Czech, Ger- man, Finnish, Romanian, Russian and Turkish) and Russian (translated …

Content truncated to 500 words