Evaluating Translation Quality and CLIR Performance of Query Sessions

Evaluating translation quality and CLIR performance of Query Sessions Xabier Saralegi1, Eneko Agirre2, Iñaki Alegria2 1Elhuyar Fundazioa, Usurbil 2IXA NLP Group, The University of the Basque Country, San Sebastian x.saralegi@elhuyar.com Abstract This paper presents the evaluation of the translation quality and Cross-Lingual Information Retrieval (CLIR) performance when using session information as the context of queries. The hypothesis is that previous queries provide context that helps to solve ambiguous translations in the current query. We tested several strategies on the TREC 2010 Session track dataset, which includes query reformulations grouped by generalization, specification, and drifting types. We study the Basque to English direction, evaluating both the translation quality and CLIR performance, with positive results in both cases. The results show that the quality of translation improved, reducing error rate by 12% (HTER) when using session information, which improved CLIR results 5% (nDCG). We also provide an analysis of the improvements across the three kinds of sessions: generalization, specification, and drifting. Translation quality improved in all three types (generalization, specification, and drifting), and CLIR improved for generalization and specification sessions, preserving the performance in drifting sessions. Keywords: Cross-Lingual Information Retrieval, Less-Resourced Languages, Machine Translation, Session-based Information Retrieval 1. Introduction The successful strategies for query translation in CLIR depend on resources such as Machine Translation systems or parallel corpora, but many languages can not rely on that kind of resources. Thus, they need other strategies based on less or more easily available resources, such as bilingual dictionaries. The translation is usually done from the language of the query to language of the target collection, mainly due to scalability reasons. Lately, some authors have underlined the importance of using session information for obtaining better rankings (Carterette et al., 2011). They claim that users often try to solve their information need by submitting more than one query, reformulating the initial query. Thus, a process regarding to an information need is often composed by several related queries. After entering an initial query, users tend to reformulate the query in different ways such as specification (e.g., qi=“scyhe” qr=”scythe mythology”), generalization (e.g., qi=“computer worms” qr=”malware”) or drifting (e.g., qi=“sun spot activity” qr=”sun spot earthquake”). Studies on web search query logs showed that half of all Web users reformulated their initial query: 52% of the users in the 1997 Excite data set and 45 % of the users in the 2001 Excite data-set (Wolfram et al., 2001). This paper studies the use of this session context in order to improve the translation quality of the queries and the corresponding retrieval process. We propose to use the previous queries of the same session in order to improve the query translation step in a CLIR system. Our hypothesis is that queries corresponding to the same session can be used as adequate additional context for improving the translation selection process. For instance, let us assume a session s={qi,qr} involving two queries in Basque: the initial query qi=”Neil Young diska“, and its reformulation qr=”Neil Young bira data” (“Neil Young album” and “Neil Young …

Content truncated to 500 words