Using a Cross-Language Information Retrieval System based on OHSUMED to Evaluate the Moses and KantanMT Statistical Machine Translation Systems

Using a Cross-Language Information Retrieval System based on OHSUMED to Evaluate the Moses and KantanMT Statistical Machine Translation Systems Nikolaos Katris1, Richard Sutcliffe2, Theodore Kalamboukis3 1Department of CSIS, University of Limerick, Limerick, Ireland 2School of CSEE, University of Essex, Wivenhoe Park, Colchester CO4 3SQ, UK 3Department of Informatics, Athens University of Economics and Business, 76 Patission Str., Athens 10434, Greece E-mail: nikos.katris@live.com, rsutcl@essex.ac.uk, tzk@aueb.gr Abstract The objective of this paper was to evaluate the performance of two statistical machine translation (SMT) systems within a cross-language information retrieval (CLIR) architecture and examine if there is a correlation between translation quality and CLIR performance. The SMT systems were KantanMT, a cloud-based machine translation (MT) platform, and Moses, an open-source MT application. First we trained both systems using the same language resources: the EMEA corpus for the translation model and language model and the QTLP corpus for tuning. Then we translated the 63 queries of the OHSUMED test collection from Greek into English using both MT systems. Next, we ran the queries on the document collection using Apache Solr to get a list of the top ten matches. The results were compared to the OHSUMED gold standard. KantanMT achieved higher average precision and F-measure than Moses, while both systems produced the same recall score. We also calculated the BLEU score for each system using the ECDC corpus. Moses achieved a higher BLEU score than KantanMT. Finally, we also tested the IR performance of the original English queries. This work overall showed that CLIR performance can be better even when BLEU score is worse. Keywords: cross-language information retrieval, statistical machine translation, Moses, KantanMT, Apache Solr, EMEA, QTLP, ECDC, OHSUMED 1. Objective Information retrieval (IR) (Grefenstette 1998) has nowadays a prominent place in our everyday lives. People of diverse backgrounds from all over the globe use search engines to look for information for nearly every imaginable human need. The word “Google” has been introduced as a verb in many natural languages, referring to the use of the famous search engine to obtain information from the World Wide Web1. Nevertheless, the Web is not the only field that requires the deployment of information retrieval systems. Searching for an email in our inbox or looking for a document within the large corporate intranet of a company containing specific keywords also represent examples of IR tasks. However, it was the rapid expansion of the World Wide Web that brought to the forefront a major challenge that IR systems needed to overcome. As the Web began to grow, the information posted started to vary in regard to language, with the amount of non-English content constantly increasing (Peters et al 2012). As a result, much information is being disregarded because it is available in a less popular language, i.e. not in English. And while most content on the Web is still available in English (over 50%2), the assumption that most users of the Internet (or most users of any IR system within an organization) have a good knowledge of the …

Content truncated to 500 words