Fine-Grained Error Analysis on English-to-Japanese Machine Translation in the Medical Domain

Fine-Grained Error Analysis on English-to-Japanese Machine Translation in the Medical Domain Takeshi Hayakawa Graduate School of Information Science and Technology, Osaka University, Osaka, Japan ASCA Corporation, Osaka, Japan hayakawa.takeshi@ist.osaka-u.ac.jp Yuki Arase Graduate School of Information Science and Technology, Osaka Un iversity, Osaka, Japan arase@ist.osaka-u.ac.jp Abstract We performed a detailed error analy- sis in domain-specific neural machine translation (NMT) for the English and Japanese language pair with fine- grained manual annotation. Despite its importance for advancing NMT technologies, research on the perfor- mance of domain-specific NMT and non-European languages has been lim- ited. In this study, we designed an error typology based on the er- ror types that were typically gener- ated by NMT systems and might cause significant impact in technical transla- tions: “Addition,” “Omission,” “Mis- translation,” “Grammar,” and “Termi- nology.” The error annotation was tar- geted to the medical domain and was performed by experienced professional translators specialized in medicine un- der careful quality control. The annotation detected 4, 912 errors on 2, 480 sentences, and the frequency and distribution of errors were analyzed. We found that the major errors in NMT were “Mistranslation” and “Ter- minology” rather than “Addition” and “Omission,” which have been reported as typical problems of NMT. Interest- ingly, more errors occurred in docu- ments for professionals compared with those for the general public. The results of our annotation work will be pub- lished as a parallel corpus with error la- © 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC-BY-ND. bels, which are expected to contribute to developing better NMT models, au- tomatic evaluation metrics, and quality estimation models. 1 Introduction We performed a manual annotation of trans- lation errors using fine-grained error typol- ogy in domain-specific neural machine transla- tion (NMT) of Japanese and English language pairs. Although several approaches have been proposed to evaluate the performance of NMT, it has been commonly presented as scores of automatic evaluation, and detailed analysis of problems in NMT is limited. Previous stud- ies (Specia et al., 2017; Kepler et al., 2019) an- notated errors in MT outputs; however, they targeted only on a general domain and Euro- pean languages. Detailed error detection is es- sential, especially in the domain-specific set- tings, where tiny mistakes, such as incorrect translation of a technical term, leads to signif- icant misunderstanding. To tackle this problem, we performed an annotation-based analysis of errors that oc- curred in NMT for a specific technical do- main. Professional translators annotated types and positions of errors that occurred in trans- lation from English to Japanese. The error typology was designed based on an existing framework, Multidimensional Quality Metrics (MQM) (Lommel et al., 2014), which was cus- tomized to our study. We selected medicine as the domain field because medical transla- tion is in growing demand in the society to enrich healthcare information, which requires highly specific domain expertise. Recent issues regarding public health, such as the pandemic of coronavirus disease 2019, highlight demands on sharing …

Content truncated to 500 words