QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5181–5190 October 12–17, 2022. 5181 QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation Sugyeong Eo1, Chanjun Park1,2, Hyeonseok Moon1, Jaehyung Seo1, Gyeongmin Kim1, Jungseob Lee1, Heuiseok Lim1∗ 1Korea University, 2Upstage {djtnrud,bcj1210,glee889,seojae777,totoro4007,omanma1928,limhseok}@korea.ac.kr chanjun.park@upstage.ai Abstract With the recent advance in neural machine translation demonstrating its importance, re- search on quality estimation (QE) has been steadily progressing. QE aims to automatically predict the quality of machine translation (MT) output without reference sentences. Despite its high utility in the real world, there remain sev- eral limitations concerning manual QE data cre- ation: inevitably incurred non-trivial costs due to the need for translation experts, and issues with data scaling and language expansion. To tackle these limitations, we present QUAK, a Korean-English synthetic QE dataset generated in a fully automatic manner. This consists of three sub-QUAK datasets QUAK-M, QUAK-P, and QUAK-H, produced through three strate- gies that are relatively free from language con- straints. Since each strategy requires no human effort, which facilitates scalability, we scale our data up to 1.58M for QUAK-P, H and 6.58M for QUAK-M. As an experiment, we quantitatively analyze word-level QE results in various ways while performing statistical analysis. Moreover, we show that datasets scaled in an efficient way also contribute to performance improvements by observing meaningful performance gains in QUAK-M, P when adding data up to 1.58M. 1 Introduction Quality estimation (QE) is the task of predicting the translation quality as a continuous value or dis- crete tags by referring to a source sentence and its machine translation (MT) output (Blatz et al., 2004; Specia et al., 2009, 2013). Since quality annotations on MT output are applied in various ways accord- ing to the granularity levels (word, sentence, docu- ment, etc.), QE research has been constantly devel- oping in recent years (Kim et al., 2017; Fomicheva et al., 2020a; Alva-Manchego et al., 2021; Ding et al., 2021b). ∗ Corresponding Author MT output Given that the Chinese authorities do not deny it , it is highly likely . pseudo-PE Given that the Chinese authorities do not deny it , chances are high . MT output tags OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK BAD OK BAD OK BAD OK BAD OK OK OK Source 중국 당국이 부인하지 않는 것으로 볼 때 가능성이 높다 . Source tags OK OK OK OK OK OK OK BAD BAD OK Alignments 0-3 1-4 2-7 3-5 3-6 4-8 5-8 6-0 7-13 8-11 8-12 9-14 Edits (1) Insertion (‘ ’→ it) (2) Substitution (chances→is) (3) Substitution (are→highly) (4) Substitution (high→likely) Table 1: An example of QUAK dataset. For the correct translation, one insertion and three substitutions are required for the MT output. Although not included in this example, if there is a missing word, a BAD tag is attached to the location of the corresponding gap token. We indicate the alignment information (Alignments) in the form of {source index}-{aligned MT …

Content truncated to 500 words