Automatic Evaluation of English-to-Korean and Korean-to-English Neural Machine Translation Systems by Linguistic Test Points

Automatic Evaluation of English-to-Korean and Korean-to-English Neural Machine Translation Systems by Linguistic Test Points Sung-Kwon Choi, Gyu-Hyeun Choi and Youngkil Kim Language Intelligence Research Group Electronics and Telecommunications Research Institute, Daejeon, Korea {choisk, choko93, kimyk}@etri.re.kr Abstract BLEU is the most well-known automatic evaluation technology in assessing the performance of machine translation systems. However, BLEU does not know which parts of the NMT translation results are good or bad. This paper describes the automatic evaluation approach of NMT systems by linguistic test points. This approach allows automatic evaluation of each linguistic test point not shown in BLEU and provides intuitive insight into the strengths and flaws of NMT systems in handling various important linguistic test points. The linguistic test points used for automatic evaluation were 58 and consisted of 630 sentences. We conducted the evaluation of two bidirectional English/Korean NMT systems. BLEUs of English-to-Korean NMT systems were 0.0898 and 0.2081 respectively, and their automatic evaluations by linguistic test points were 58.35% and 77.31%, respectively. BLEUs of Korean-to-English NMT systems were 0.3939 and 0.4512 respectively, and their automatic evaluations by linguistic test points were 33.10% and 40.47%, respectively. This means that the automatic evaluation approach by linguistic test points has similar results as BLEU assessment. According to automatic evaluation by linguistic test points, we know that both English-to-Korean NMT systems and Korean-to-English NMT systems have strengths in polysemy translations, but has flaws in style translations and translations of sentences with complex syntactic structures. 1 Introduction Currently, the performance of NMT system is rapidly advancing. Its performance proved to be superior to that of SMT (Bojar et. al., 2016). The NMT is also applied to both English-to-Korean and Korean-to-English machine translation systems for commercial service. The most well- known machine translation evaluation technique in assessing the performance of NMT systems is BLEU (Papineni et. al., 2002). It is strength of BLEU that automatic scores for MT output can be provided in cases where there are existing reference translations by calculating similarity between the MT output and the references. Faults of BLEU are that it does not provide insight into the specific nature of problems encountered in the translation output and scores are tied to the particularities of the reference translations (Lommel et. al., 2014). By BLEU, developers and users cannot identify which part of the NMT translation result is vulnerable. We propose an automatic evaluation approach of neural machine translation systems by linguistic test points. Instead of assigning a general score to an NMT system we conduct an automatic evaluation by each linguistic test point not shown in BLEU. This automatic evaluation approach of this paper can give developers an intuitive insight into the strengths and flaws of NMT systems. Also, the automatic evaluation method by linguistic test points, which is not like BLEU, may provide objective evaluation even without reference sentences. Section 2 describes existing studies related to the automatic evaluation approach by linguistic test points. Section 3 introduces the design of test set PACLIC 32 115 32nd Pacific Asia Conference on …

Content truncated to 500 words