Better Quality Estimation for Low Resource Corpus Mining

Findings of the Association for Computational Linguistics: ACL 2022, pages 533 - 543 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Better Quality Estimation for Low Resource Corpus Mining Muhammed Yusuf Kocyigit Boston University kocyigit@bu.edu Jiho Lee Boston University jiholee@bu.edu Derry Wijaya Boston University wijaya@bu.edu Abstract Quality Estimation (QE) models have the po- tential to change how we evaluate and maybe even train machine translation models. How- ever, these models still lack the robustness to achieve general adoption. We show that State- of-the-art QE models, when tested in a Par- allel Corpus Mining (PCM) setting, perform unexpectedly bad due to a lack of robustness to out-of-domain examples. We propose a combi- nation of multitask training, data augmentation and contrastive learning to achieve better and more robust QE performance. We show that our method improves QE performance signifi- cantly in the MLQE challenge and the robust- ness of QE models when tested in the Parallel Corpus Mining setup. We increase the accuracy in PCM by more than 0.80, making it on par with state-of-the-art PCM methods that use mil- lions of sentence pairs to train their models. In comparison, we use thousand times less data, 7K parallel sentences in total, and propose a novel low resource PCM method. 1 Introduction The Quality Estimation (QE) task aims to model human perception of translation quality and pre- dict the quality score an expert would give to a translation using only the source sentence and the translation. This requires the QE model to rep- resent the cross-lingual similarity between source and hypothesis sentences while incorporating dif- ferent features of the hypothesis sentence such as fluency, grammaticality and adequacy1. Human evaluations of machine translation are costly and time-consuming for a large-scale text dataset. References to evaluate machine transla- tion performance are not readily available in many cases, especially in low-resource languages. Even 1Fluency measures whether a translation is fluent, regard- less of the correct meaning, while Adequacy measures whether the translation conveys the correct meaning, even if the trans- lation is not fully fluent (Snover et al., 2009) if they do exist, they often assume a single, unique answer for correct translations, causing bias in the evaluation. Thus, it is academically and profession- ally of paramount importance to further develop reliable Quality Estimation metrics, which can ulti- mately eliminate the need for references and have unlimited potential for practical applications in ma- chine translations. Parallel Corpus Mining (PCM) is another criti- cal task that can enable the creation of high-quality parallel data and reduce the need for considerable human effort. These mined parallel corpora could especially be helpful in low resource languages. On the other hand, current PCM methods require large amounts of parallel data, which creates a para- doxical loop that only large companies can break. Quality estimation is uniquely linked with PCM since what makes a good translation most of the time makes a correct parallel too. Considering the similarity in the underlying goals of these two tasks, we expect models that can …

Content truncated to 500 words