An Investigation of Machine Translation Evaluation Metrics in Cross-lingual Question Answering

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 442–449, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. An Investigation of Machine Translation Evaluation Metrics in Cross-lingual Question Answering Kyoshiro Sugiyama, Masahiro Mizukami, Graham Neubig, Koichiro Yoshino, Sakriani Sakti, Tomoki Toda, Satoshi Nakamura Graduate School of Information Science Nara Institute of Science and Technology Takayamacho 8916-5, Ikoma, Nara {sugiyama.kyoshiro.sc7, neubig}@is.naist.jp Abstract Through using knowledge bases, ques- tion answering (QA) systems have come to be able to answer questions accurately over a variety of topics. However, knowl- edge bases are limited to only a few ma- jor languages, and thus it is often nec- essary to build QA systems that answer questions in one language based on an in- formation source in another (cross-lingual QA: CLQA). Machine translation (MT) is one tool to achieve CLQA, and it is intu- itively clear that a better MT system im- proves QA accuracy. However, it is not clear whether an MT system that is better for human consumption is also better for CLQA. In this paper, we investigate the re- lationship between manual and automatic translation evaluation metrics and CLQA accuracy by creating a data set using both manual and machine translations and per- form CLQA using this created data set.1 As a result, we find that QA accuracy is closely related with a metric that consid- ers frequency of words, and as a result of manual analysis, we identify 3 factors of translation results that affect CLQA accu- racy. 1 Introduction Question answering (QA) is the task of searching for an answer to question sentences using some variety of information resource. Generally, docu- ments, web pages, or knowledge bases are used as these information resources. When the language of the question differs from the language of the in- formation resource, the task is called cross-lingual question answering (CLQA) (Magnini et al., 2004; 1All data used in the experiments will be released upon publishing of the paper. Sasaki et al., 2007). Machine translation (MT) is one of the most widely used tools to achieve CLQA (Mori and Kawagishi, 2005; Fujii et al., 2009; Kettunen, 2009).2 In the realm of monolingual question answer- ing, recent years have seen a large increase in the use of structured knowledge bases such as Free- base (Bollacker et al., 2008), as they allow for accurate answering of questions over a variety of topics (Frank et al., 2007; Cai and Yates, 2013). However, knowledge bases are limited to only a few major languages. Thus, CLQA is particularly important for QA using knowledge bases. In contrast to the CLQA situation, where an MT system is performing translation for a downstream system to consume, in standard translation tasks the consumer of results is a human (Matsuzaki et al., 2015). In this case, it is important to define an evaluation measure which has high correlation with human evaluation, and the field of MT met- rics has widely studied which features of MT re- sults are correlated with human evaluation, and …

Content truncated to 500 words