Multilingual Machine Translation Evaluation Metrics Fine-tuned on Pseudo-Negative Examples for WMT 2021 Metrics Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1049–1052 November 10–11, 2021. ©2021 Association for Computational Linguistics 1049 Multilingual Machine Translation Evaluation Metrics Fine-tuned on Pseudo-Negative Examples for WMT 2021 Metrics Task Kosuke Takahashi1, Yoichi Ishibashi1, Katsuhito Sudoh1,2, Satoshi Nakamura1 1 Nara Institute of Science and Technology 2 PRESTO, Japan Science and Technology Agency {takahashi.kosuke.th0, ishibashi.yoichi.ir3, sudoh, s-nakamura}@is.naist.jp Abstract This paper describes our submission to the WMT2021 shared metrics task. Our metric is operative to segment-level and system-level translations. Our belief toward a better metric is to detect a significant error that cannot be missed in the real practice cases of evaluation. For that reason, we used pseudo-negative ex- amples in which attributes of some words are transferred to the reversed attribute words, and we build evaluation models to handle such se- rious mistakes of translations. We fine-tune a multilingual largely pre-trained model on the provided corpus of past years’ metric task and fine-tune again further on the synthetic nega- tive examples that are derived from the same fine-tune corpus. From the evaluation results of the WMT21’s development corpus, fine- tuning on the pseudo-negatives using WMT15- 17 and WMT18-20 metric corpus achieved a better Pearson’s correlation score than the one fine-tuned without negative examples. Our submitted models are named C-SPEC (Cross- lingual Sentence Pair Embedding Concatena- tion) and C-SPECpn, are the plain model us- ing WMT18-20 and the one additionally fine- tuned on negative samples, respectively. 1 Introduction Recent studies of automatic evaluation is mostly based on the family models of BERT (Devlin et al., 2019). BERTscore (Zhang et al., 2020), BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020) have shown a strong correlation with human judge- ment scores. However, we reported in our previous study (Takahashi et al., 2020), it is hard for BERT based metrics to correctly evaluate the translation errors that are annotated with low Direct Assess- ment (DA) score. Upon the problems of evaluating poor quality translations, Sudoh et al. (2021) has attempted to solve the problem by creating a different human annotation set and corpus. Compared to DA, their idea is to make a clear definition of critical transla- tion errors and let models learn the critical errors that can cause a serious misunderstanding. Following the idea, we used pseudo-negative ex- amples to train the evaluation model. Since, empiri- cally, the cases of the evaluation failure happens fre- quently with the nouns translation errors, we gen- erated pseudo-negative sentences by transferring the attribute of nouns with Word Attribute Transfer (Ishibashi et al., 2020). This system is based on our previous work (Takahashi et al., 2020), with an ex- tension with fine-tuning with the pseudo-negative examples. 2 Related Work BERTscore (Zhang et al., 2020), BERT regres- sor (Shimanaka et al., 2019), BLEURT (Sellam et al., 2020), and COMET (Rei et al., 2020) are applications of BERT to the machine translation evaluation. BERTscore measures the similarity of reference and hypothesis translation by the cosine- similarity of the token embeddings for each to- ken …

Content truncated to 500 words