Levenshtein Training for Word-level Quality Estimation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6724–6733 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6724 Levenshtein Training for Word-level Quality Estimation Shuoyang Ding†∗ Marcin Junczys-Dowmunt‡ Matt Post†‡ Philipp Koehn† † Center for Language and Speech Processing, Johns Hopkins University ‡ Microsoft {dings, phi}@jhu.edu {marcin.junczysdowmunt, mattpost}@microsoft.com Abstract We propose a novel scheme to use the Levenshtein Transformer to perform the task of word-level quality estimation. A Leven- shtein Transformer is a natural fit for this task: trained to perform decoding in an iterative manner, a Levenshtein Transformer can learn to post-edit without explicit supervision. To further minimize the mismatch between the translation task and the word-level QE task, we propose a two-stage transfer learning proce- dure on both augmented data and human post- editing data. We also propose heuristics to con- struct reference labels that are compatible with subword-level finetuning and inference. Re- sults on WMT 2020 QE shared task dataset show that our proposed method has superior data efficiency under the data-constrained set- ting and competitive performance under the unconstrained setting. 1 Introduction Quality estimation (QE) is the task of estimating the quality of translation without access to a human- generated reference. Most recent advances on qual- ity estimation (Rei et al., 2020; Thompson and Post, 2020a; Ranasinghe et al., 2020, inter alia) focus on estimating the quality of translation on either the corpus or segment-level. However, in practice, the end-users of machine translation (MT) often call for quality signals on more fine-grained level—the level of individual words in a translation. Such sig- nals are not only useful for more fine-grained triage of translation quality, but also open up the poten- tial for targeted post-processing and faster human post-editing. To automatically assess the quality of transla- tions, it is natural to consider starting from a ma- chine translation model, which has already ac- quired the translation knowledge. However, it is not ∗ Shuoyang Ding had a part-time affiliation with Mi- crosoft at the time of this work. (a) Vaccinations put ends to the pandemic ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Vaccinations put an end to the pandemic ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? OK OK OK BAD BAD OK OK OK OK OK OK target post-edit TER (b) Vaccinations put ends to the pandemic ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? NO NO YES NO NO NO prev. target deletion mask ins. word ins. 0 2 0 0 Vaccinations put an end to the pandemic Figure 1: Figure (a) shows an example of TER-style edit tags used as reference for word-level quality esti- mation task. Figure (b) shows a series of hypothetical Levenshtein Transformer edits that generates the same sequence from the target input. The similarity of these edit operations motivates the study in this …

Content truncated to 500 words