CHARCUT: Human-Targeted Character-Based MT Evaluation with Loose Differences

CHARCUT: Human-Targeted Character-Based MT Evaluation with Loose Differences Adrien Lardilleux*, Yves Lepage** *Fujitsu Technology Solutions, Luxembourg *DGT, European Commission, Luxembourg **IPS, Waseda University, Japan adrien.lardilleux@ext.ec.europa.eu, yves.lepage@waseda.jp Abstract We present CHARCUT, a character-based machine transla- tion evaluation metric derived from a human-targeted seg- ment difference visualisation algorithm. It combines an it- erative search for longest common substrings between the candidate and the reference translation with a simple length- based threshold, enabling loose differences that limit noisy character matches. Its main advantage is to produce scores that directly reflect human-readable string differences, mak- ing it a useful support tool for the manual analysis of MT output and its display to end users. Experiments on WMT16 metrics task data show that it is on par with the best “un- trained” metrics in terms of correlation with human judge- ment, well above BLEU and TER baselines, on both system and segment tasks. 1. Introduction A large number of metrics have been proposed in the past years for the task of objective evaluation of Ma- chine Translation. To this day, trained or combined met- rics (e.g. BEER [1], DPMFCOMB [2], UOW.REVAL [3], COBALTF [4], among others), generally attain top results in terms of average correlation with human judgement, as was concluded in recent WMT conferences [5, 6]. On the other hand, endogenous metrics present the advantage of versatility, the most widely used remaining BLEU [7] and TER [8]. Such versatility is crucial in envi- ronments where MT systems are built and tuned to support numerous languages, some of which having little resources available. Among those metrics, character-based ones have re- ceived more and more interest, starting from BLEU in char- acters [9, 10] to the recent CHRF [11] and CharacTER [12]. Operating at the character level is all the more important as MT systems working at a sub-word level are getting more widely used, e.g. with segmentation schemes like Byte Pair Encoding in Neural MT [13]. Since character-based met- rics can implicitly account for sub-word linguistic phenom- ena, they have shown to correlate much better with human judgements than BLEU and TER—sometimes by tremen- dous margins [5, 6]. Nowadays, such metrics seem to be safe for use as drop-in replacements for BLEU. Another key aspect of MT evaluation is the display of MT output and its comparison with human references. Highlight- ing differences between a candidate and a reference transla- tion is a standard feature of many translation interfaces (e.g. MateCat’s edit log [14] or SDL Studio’s SDLXLIFF Com- pare1). Basically, any string comparison tool operating at a sub-segment level could be used to that end. The potential associated scores, however, typically simple word or charac- ter match percentages, may not reflect all aspects expected from a MT metric. One could just replace it with another metric, but inconsistencies between score and visible differ- ences would ensue [15], which might be confusing for non- specialists. Advanced analysis environments proposing mul- tiple measures along with word-based highlighting, such as Asiya [16] or MT-ComparEval [17], among others, …

Content truncated to 500 words