The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3497–3502, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3497 The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval Constantine Lignos1, Daniel Cohen2, Yen-Chieh Lien2, Pratik Mehta1, W. Bruce Croft2, Scott Miller1 1Information Sciences Institute, University of Southern California 2Center for Intelligent Information Retrieval, University of Massachusetts Amherst lignos@brandeis.edu {dcohen,ylien,psmehta,croft}@cs.umass.edu, smiller@isi.edu Abstract When performing cross-language informa- tion retrieval (CLIR) for lower-resourced lan- guages, a common approach is to retrieve over the output of machine translation (MT). How- ever, there is no established guidance on how to optimize the resulting MT-IR system. In this paper, we examine the relationship between the performance of MT systems and both neu- ral and term frequency-based IR models to identify how CLIR performance can be best predicted from MT quality. We explore perfor- mance at varying amounts of MT training data, byte pair encoding (BPE) merge operations, and across two IR collections and retrieval models. We find that the choice of IR col- lection can substantially affect the predictive power of MT tuning decisions and evaluation, potentially introducing dissociations between MT-only and overall CLIR performance. 1 Introduction For cross-language information retrieval scenar- ios involving queries in a higher-resourced lan- guage and documents in a lower-resourced lan- guage, direct training of a cross-language IR sys- tem (Litschko et al., 2018; Sasaki et al., 2018; Vuli´c and Moens, 2015) is typically infeasible due to in- sufficient data in the document language. A practical solution is to use machine transla- tion to translate documents into the language of the queries, enabling the use of a traditional monolin- gual IR system trained on the higher-resourced lan- guage. Prior work (Pecina et al., 2014; Nikoulina et al., 2012) demonstrates the effectiveness of us- ing an MT system for query translation within a CLIR framework, but increases in MT performance measured by BLEU (Papineni et al., 2002) do not necessarily correlate with IR performance. Tak- ing a document translation approach, Hieber and Riezler (2015) show that by enforcing a bag of words constraint on the MT decoding process, a translated document can provide richer input to an MT-IR system. However, these prior works do not fully investigate the effectiveness of MT metrics in predicting performance of a downstream IR model. In this paper, we explore the consequences of per- forming information retrieval over machine trans- lation and address three questions regarding the performance of the resulting system. First, when making MT tuning decisions such as selecting the number of BPE merge operations, will a value that improves MT performance generally increase IR performance? Second, when evaluating MT perfor- mance, can standard metrics be adapted to better correlate with downstream IR performance? Fi- nally, what is the general relationship between the performance of MT and a downstream IR task, and how is it affected by the choice of retrieval model? 2 Experiment Design …

Content truncated to 500 words