Partial Could Be Better than Whole. HW-TSC 2022 Submission for the Metrics Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 549 - 557 December 7-8, 2022 ©2022 Association for Computational Linguistics Partial Could Be Better Than Whole. HW-TSC 2022 Submission for the Metrics Shared Task Yilun Liu∗, Xiaosong Qiao∗, Zhanglin Wu, Chang Su, Min Zhang, Yanqing Zhao, Song Peng, Shimin Tao, Hao Yang, Ying Qin, Jiaxin Guo, Minghan Wang, Yinglu Li, Peng Li, Xiaofeng Zhao Huawei Translation Services Center, Beijing, China {liuyilun3, qiaoxiaosong, wuzhanglin2, suchang8, zhangmin186, zhaoyanqing, pengsong2, taoshimin, yanghao30, qinying, guojiaxin, wangminghan} @huawei.com Abstract In this paper, we present the contribution of HW-TSC to WMT 2022 Metrics Shared Task. We propose one reference-based metric, HWTSC-EE-BERTScore*, and four reference- free metrics including HWTSC-Teacher-Sim, HWTSC-TLM, KG-BERTScore and CROSS- QE. Among these metrics, HWTSC-Teacher- Sim and CROSS-QE are supervised, whereas HWTSC-EE-BERTScore*, HWTSC-TLM and KG-BERTScore are unsupervised. We use these metrics in the segment-level and system- level tracks. Overall, our systems achieve strong results for all language pairs on previ- ous test sets and a new state-of-the-art in many sys-level case sets. 1 Introduction Due to the expensive cost of manual evaluation, automatically evaluating the outputs of transla- tion systems is critically important in the field of machine translation (MT) (Freitag et al., 2021a). Therefore, a lot of automatic metrics have been pro- posed to approach this task. According to whether the reference sentences are required or not, the met- rics are categorized into two classes: (1) reference- based metrics like BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), BERTScore (Zhang et al., 2020) and BLEURT (Sellam et al., 2020), which evaluate the hypothesis by referring to the golden reference; (2) reference-free metrics like YiSi-2 (Lo, 2019) and COMET-QE (Rei et al., 2020, 2021), which are also referred as quality esti- mation (QE). These metrics estimate the quality of hypothesis only based one source sentences with- out using references. In this paper, we present the contribution of HW-TSC to the WMT 2022 Shared Task on Met- rics. We participated in the segment-level and system-level tracks with 1 reference-based metric (HWTSC-EE-BERTScore*) and 4 reference-free ∗ equal contribution metrics (HWTSC-Teacher-Sim, HWTSC-TLM, KG-BERTScore and CROSS-QE). Details of our metrics are illustrated in Table 1. HWTSC-EE-BERTScore* (Entropy Enhanced Metrics)is built upon existing metrics, aiming to achieve a more balanced system-level rating by as- signing weights to segment-level scores produced by backbone metrics. The weights are determined by the difficulty of a segment, which is related to the entropy of a hypothesis-reference pair. A trans- lation hypothesis with a significantly high entropy value is considered difficult and receives a large weight in aggregation of EE-Metrics’ system-level scores. HWTSC-Teacher-Sim is a supervised reference- free metric with the framework of BERTScore (Zhang et al., 2020), which is obtained by fine- turning the multilingual Sentence-BERT model (Reimers and Gurevych, 2019, 2020a). Both the un- supervised TearcherSim (Yang et al., 2022b,a) and the implicit multilingual word embedding align- ment (Zhang et al., 2022b) have shown that the pre- tained multilingual Sentence-BERT model is very effective for both reference-based and reference- …

Content truncated to 500 words