Improving Semantic Similarity Calculation of Japanese Text for MT Evaluation

Improving Semantic Similarity Calculation of Japanese Text for MT Evaluation Yuki Tanahashi WANTS Inc. Nagoya, Japan tanahashi236@wantsinc.jp Eiko Yamamoto Gifu Shotoku Gakuen University Gifu, Japan eiko@gifu.shotoku.ac.jp Kyoko Kanzaki Toyohashi University of Technology Toyohashi, Japan kanzaki@cite.tut.ac.jp Hitoshi Isahara Toyohashi University of Technology Toyohashi, Japan isahara@tut.jp Abstract In recent years, the quality of machine transla- tion has significantly improved, and it is con- sidered that translation of a novel becomes possible. However, when performing transla- tion for novels by a machine translation, it needs an evaluation method that performs com- parison using a vector of distributed represen- tations. BERTScore is one of the methods for performing automatic evaluation using a dis- tributed representation. In this research, we ex- amined the optimal setting for applying the BERTScore to the evaluation of translations of Japanese novels, and improved the method by introducing penalties for named entities based on idf values calculated from large corpora. The introduction of the penalty has made it possible to mitigate the false matching of per- sonal names caused by distributed representa- tion. We verified the method by calculating the Pearson correlation between the modified BERTScore and human-rated scores. Further- more, we set four BERT models and two kinds of corpora to calculate idf value, and investi- gated which setting is most suitable for evalua- tion of novel translation. As a result, the setting with the model based on novel corpus, the idf based novel corpus and the penalty had the highest correlation with human-rated scores. 1 Introduction In recent years, machine translation quality has dra- matically improved due to the development of neu- ral translation models that utilize deep learning, such as the sequence transformation model (Sutskever et al., 2014) and the attention model (Dzmitry et al., 2015. Luong et al., 2015), which is an application of the attention mechanism, and im- proved computer performance. Due to these im- provements, not only documents consisting of formal expressions such as patent sentences and ac- ademic papers that have been fixed to some extent but also informal expressions such as novels and colloquial expressions could be machine translated. However, previous research has revealed that prob- lems that have not been considered as important in machine translation research so far have a great in- fluence on the learning and results of novel transla- tion. Among them, the variety of text expressions is a serious problem. The problem is that when the au- thor or translator in a novel is different, or even if the same author/translator has a different story speaker, an English sentence is translated into a Jap- anese sentence with a distinctly different translation but with a similar meaning. Specific examples are shown below. When a language resource (corpus) containing such parallel translations is used as learning data in neu- ral translation that receives the whole sentence as input and learns so as to maximize the likelihood that a correct word sequence will be output, learning becomes difficult and the output of the translation system becomes unstable. In addition, different …

Content truncated to 500 words