The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 904–910 November 10–11, 2021. ©2021 Association for Computational Linguistics 904 The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task Shuoyang Ding†∗ Marcin Junczys-Dowmunt‡ Matt Post†‡ Christian Federmann‡ Philipp Koehn† † Center for Language and Speech Processing, Johns Hopkins University ‡ Microsoft {dings, phi}@jhu.edu {marcin.junczysdowmunt, mattpost, chrife}@microsoft.com Abstract This paper presents the JHU-Microsoft joint submission for WMT 2021 quality estimation shared task. We only participate in Task 2 (post-editing effort estimation) of the shared task, focusing on the target-side word-level quality estimation. The techniques we experi- mented with include Levenshtein Transformer training and data augmentation with a combi- nation of forward, backward, round-trip trans- lation, and pseudo post-editing of the MT out- put. We demonstrate the competitiveness of our system compared to the widely adopted OpenKiwi-XLM baseline. Our system is also the top-ranking system on the MT MCC met- ric for the English-German language pair. 1 Introduction In the machine translation (MT) literature, qual- ity estimation (QE) refers to the task of evaluating the translation quality of a system without using a human-generated reference. There are several different granularities as to the way those quality labels or scores are generated. Our participation in the WMT21 quality estimation shared task fo- cuses specifically on the word-level quality labels (word-level subtask of Task 2), which are helpful for both human (Lee et al., 2021) and automatic (Lee, 2020a) post-editing of translation outputs. The task asks the participant to predict one binary quality label (OK/BAD) for each target word and each gap between target words, respectively.1 Our approach closely follows our contemporary work (Ding et al., 2021), which focuses on en-de and en-zh language pairs tested in the 2020 ver- sion of the shared task. The intuition behind our idea is that translation knowledge is very useful for predicting word-level quality labels of translations. ∗ Shuoyang Ding had a part-time affiliation with Mi- crosoft at the time of this work. 1While there is another sub-task for predicting source-side quality labels, we do not participate in that task. However, usage of machine translation models is limited in the previous work mainly due to (1) the difficulties in using both the left and right context of an MT word to be evaluated; (2) the difficulties in making the word-level reference labels compatible with subword-level models; and (3) the difficul- ties in enabling translation models to predict gap labels. To resolve these difficulties, we resort to Levenshtein Transformer (LevT, Gu et al., 2019), a model architecture designed for non-autoregressive neural machine translation (NA-NMT). Because of its iterative inference procedure, LevT is capable of performing post-editing on existing translation output even just trained for translation. To further improve the model performance, we also propose to initialize the encoder and decoder of the LevT model with those from a massively pre-trained mul- tilingual NMT model (M2M-100, Fan et al., 2020). Starting from a LevT translation model, we then perform a two-stage finetuning process to adapt the model from translation …

Content truncated to 500 words