Quality Scoring of Source Words in Neural Translation Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10683 - 10691 December 7-11, 2022 ©2022 Association for Computational Linguistics Quality Scoring of Source Words in Machine Translations Priyesh Jain priyeshjbp@gmail.com Sunita Sarawagi sunita@iitb.ac.in Indian Institute of Technology Bombay, India Tushar Tomar tomartushar@cse.iitb.ac.in Abstract Word-level quality scores on input source sen- tences can provide useful feedback to an end- user when translating into an unfamiliar target language. Recent approaches either require training custom models on synthetic data or re- peatedly invoking the translation model. We propose a simple approach based on compar- ing probabilities from two language models. The basic premise of our method is to reason how well each source word is explained by the generated translation as against the preceding source language words. Our approach provides between 2.2 and 27.1 higher F1 score and is significantly faster than state of the art meth- ods on three language pairs. Also, our method does not require training any new model. We release a public dataset on word omissions and mistranslations on a new language pair. 1 1 Introduction Neural Machine Translations exhibit human- like fluency and accuracy over many language pairs (Läubli et al., 2018), and are increasingly get- ting embedded in several end-user applications. A hindrance to their deployment in safety-critical ap- plications such as healthcare and diplomacy, is that unlike human translators, a machine translation sys- tem does not provide reliable feedback when parts of a source sentence are misrepresented in the gen- erated translation. Recent studies have discovered that NMT sentences while fluent are often incon- sistent with the source (Maynez et al., 2020; Wang and Sennrich, 2020). On the target side, calibrated confidence with the generated output is a standard way to give quality feedback to the user (Kumar and Sarawagi, 2019; Lu et al., 2022). However, when a user is unfamiliar with the target language, word-level confidence in the generated translation is not actionable. We instead propose that words in 1Our code is available at this link https://github.com/ jain-priyesh/target-lift.git the source sentence be assigned indicators of the loss of fidelity, so end-users could reformulate the input for possibly better translation, much like the way a human translator could ask for clarification on misunderstood words. As a step in this direction, we attempt to solve the following problem: Given a source sentence x with words x1, . . . xn and its translation ˆy from a translation model, assign each word xi in x a score q(xi) that when low indicates that the word xi is either omitted or mistranslated in ˆy. Figure 1 shows an example with a source English sentence and its translation to German. The words in the source sentence that are omitted or mis-translated are assigned relatively small scores. Two categories of approaches have been ex- plored so far. The first category (Zhou et al., 2021) train a supervised model to score words using synthetically inserted errors in parallel translation datasets. Vamvas and Sennrich (2022) show that such …

Content truncated to 500 words