Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method

Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method Yutong Shao1, Rico Sennrich2, Bonnie Webber2, Federico Fancellu2 1School of Electronic Engineering and Computer Science, Peking University 5 Yiheyuan Road, Haidian District, Beijing, China sythello@pku.edu.cn 2School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, United Kingdom, rico.sennrich@ed.ac.uk, bonnie@inf.ed.ac.uk, s1260346@sms.ed.ac.uk Abstract Idiom translation is a challenging problem in machine translation because the meaning of idioms is non-compositional, and a literal (word-by-word) translation is likely to be wrong. In this paper, we focus on evaluating the quality of idiom translation of MT systems. We introduce a new evaluation method based on an idiom-specific blacklist of literal translations, based on the insight that the occurrence of any blacklisted words in the translation output indicates a likely translation error. We introduce a dataset, CIBB (Chinese Idioms Blacklists Bank), and perform an evaluation of a state-of-the-art Chinese→English neural MT system. Our evaluation confirms that a sizable number of idioms in our test set are mistranslated (46.1%), that literal translation error is a common error type, and that our blacklist method is effective at identifying literal translation errors. Keywords: Chinese-English machine translation, evaluation, idiom translation, blacklist method, CIBB dataset 1. Introduction Idioms are a special figure of speech that are non- compositional and non-literal, though occasionally share surface realizations with literal language uses (Salton et al., 2014b). Idioms are considered highly problematic for a wide variety of NLP tasks (Sag et al., 2002). This belief also holds true for machine translation, because MT sys- tems often make the assumption that meaning is composi- tional, which is not true for idioms. The compositionality assumption leads to literal translation errors, the word-by- word translation of idioms, resulting in a translation that is confusing and not understandable. Therefore, idiom trans- lation is a hard problem in MT and has attracted consider- able research interest (Cap et al., 2015; Salton et al., 2014b; Anastasiou, 2010). Given the difficulty of idiom translation in MT, it would be helpful to have a method to evaluate idiom translation per- formance. There is a wide range of methods for evaluating the performance of MT systems, but none of them are satis- factory for the targeted evaluation of idiom translation. The most straightforward method is human evaluation. While human evaluation is highly valuable, it is desirable to de- velop complementary automatic methods that are low-cost and fast, thus allowing for more rapid and frequent feed- back cycles. Popular automatic MT metrics such as BLEU (Papineni et al., 2002) are inexpensive, but are unsuitable for a targeted evaluation. This paper tries to fill this gap by presenting a method to as- sess the quality of idiom translations. We introduce a new method called “blacklist method” for performance evalua- tion on idioms, which is based on the intuition that a literal translation of the components of the idiom is likely to be wrong, and easy to spot by defining a blacklist of words that indicate a likely literal translation error. We perform a case study …

Content truncated to 500 words