Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4959 - 4970 December 7-11, 2022 ©2022 Association for Computational Linguistics Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? Pei Zhang ∗, Baosong Yang, Haoran Wei, Dayiheng Liu, Kai Fan, Luo Si and Jun Xie Alibaba Group Inc. {xiaoyi.zp,yangbaosong.ybs,funan.whr,qingjing.xj}@alibaba-inc.com Abstract Neural machine translation (NMT) is often crit- icized for failures that happen without aware- ness. The lack of competency awareness makes NMT untrustworthy. This is in sharp contrast to human translators who give feedback or con- duct further investigations whenever they are in doubt about predictions. To fill this gap, we propose a novel competency-aware NMT by extending conventional NMT with a self- estimator, offering abilities to translate a source sentence and estimate its competency. The self- estimator encodes the information of the de- coding procedure and then examines whether it can reconstruct the original semantics of the source sentence. Experimental results on four translation tasks demonstrate that the pro- posed method not only carries out translation tasks intact but also delivers outstanding per- formance on quality estimation. Without de- pending on any reference or annotated data typically required by state-of-the-art metric and quality estimation methods, our model yields an even higher correlation with human quality judgments than a variety of aforemen- tioned methods, such as BLEURT, COMET, and BERTScore. Quantitative and qualitative analyses show better robustness of competency awareness in our model.1 1 Introduction With the exploitation of large amounts of train- ing data, neural machine translation (NMT) mod- els (Bahdanau et al., 2015; Vaswani et al., 2017) have shown promising quality. However, NMT often makes mistakes, especially on outliers such as out-of-domain samples, input noises, and low- frequency words, on which models are potentially not trained or evaluated (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018; Müller et al., 2020). Unlike human translators who are generally aware ∗*Corresponding author. 1Code and test sets are available at: https://github.com/xiaoyi0814/CANMT. Method Input → Output Human-Assist MT (src) → (trans) N/A Metric (src, ref, trans) → (quality) ✓ QE (src, trans) → (quality) ✓ CANMT (src) → (trans, quality) × Table 1: Comparison among Machine Translation (MT), Automatic Translation Evaluation (Metric), Quality Es- timation (QE), and our Competency-Aware Neural Ma- chine Translation (CANMT). ‘Input→Output’ indicates the input and output for each task, where src, ref, trans, and quality represent source, reference, translation, and quality score respectively. ‘Human-Assist’ denotes whether quality evaluation depends on human-annotated references or samples with quality scores for training. of their expertise and weaknesses, NMT lacks com- petency awareness, which limits the trustworthi- ness when it is widely adopted. Consequently, it is necessary to equip NMT models with good aware- ness of their translation competency, i.e. the ability to give a translation along with the quality score of the translation. With good awareness of its com- petency, NMT can then take conservative actions (such as human intervention) on low-quality trans- lations, and request fewer manual inspections for high-quality ones. In order to …

Content truncated to 500 words