cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1014–1023 November 10–11, 2021. ©2021 Association for Computational Linguistics 1014 cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE Lifeng Han1, Irina Sorokina2, Gleb Erofeev2, and Serge Gladkoff2 1 ADAPT Research Centre, DCU, Ireland 2 Logrus Global, Translation & Localization lifeng.han@adaptcentre.ie gleberof, irina.sorokina, serge.gladkoff@logrusglobal.com Abstract Human evaluation has always been expen- sive while researchers struggle to trust the automatic metrics. To address this, we pro- pose to customise traditional metrics by tak- ing advantages of the pre-trained language models (PLMs) and the limited available hu- man labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python version we developed (ported) which achieved the automatic tuning of the weight- ing parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLE- POR) which uses Optuna hyper-parameter op- timisation framework to fine-tune hLEPOR weighting parameters towards better agree- ment to pre-trained language models (using LaBSE) regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards profes- sional human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The ex- perimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LaBSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU (data available at https://github. com/poethan/cushLEPOR). Official re- sults show that our submissions win three lan- guage pairs including English-German and Chinese-English on News domain via cushLE- POR(LM) and English-Russian on TED do- main via hLEPOR. 1 Introduction Machine Translation (MT) is a rapidly developing research field that plays an important role in NLP area. MT started from 1950s as one of the earli- est artificial intelligence (AI) research topics and gained a large improvement in the output quality in large resourced language pairs after the introduc- tion of Neural MT (NMT) in recent years (Kalch- brenner and Blunsom, 2013; Cho et al., 2014; Bah- danau et al., 2014). However, the challenge still remains in achieving human parity of MT output (Han et al., 2021a). Thus MT evaluation (MTE) continues to play an important role in aiding MT development from the aspects of timely and high quality evaluations, as well as reflecting the trans- lation errors that MT systems can take advantages of for further improvement (Han et al., 2021b). On one hand, human evaluations have long been criti- cised as expensive and unrepeatable. Furthermore, the inter- and intra-agreement levels from Human raters may struggle to achieve a consistent and reliable score, unless done in rigour with highly trained and skilled evaluators (Alekseeva et al., 2021). On the other hand, even though researchers have claimed that the automatic evaluation met- rics have reached much better performances in the category of system level evaluations of MT out- puts, with high correlation to human judgements, the segment level performance is still a large gap from human experts’ expectation …

Content truncated to 500 words