Improving Translation of Queries with Infrequent Unknown Abbreviations and Proper Names

Computational Linguistics and Chinese Language Processing Vol. 13, No. 1, March 2008, pp. 91-120 91 © The Association for Computational Linguistics and Chinese Language Processing [Received June 30, 2007; Revised October 8, 2007; Accepted January 12, 2008] Improving Translation of Queries with Infrequent Unknown Abbreviations and Proper Names Wen-Hsiang Lu∗, Jiun-Hung Lin+, and Yao-Sheng Chang* Abstract Unknown term translation is important to CLIR and MT systems, but it is still an unsolved problem. Recently, a few researchers have proposed several effective search-result-based term translation extraction methods which explore search results to discover translations of frequent unknown terms from Web search results. However, many infrequent unknown terms, such as abbreviations and proper names (or named entities), and their translations are still difficult to be obtained using these methods. Therefore, in this paper we present a new search-result-based abbreviation translation method and a new two-stage hybrid translation extraction method to solve the problem of extracting translations of infrequent unknown abbreviations and proper names from Web search results. In addition, to efficiently apply name transliteration techniques to mitigate the problems of proper name translation, we propose a mixed-syllable-mapping transliteration model and a Web-based unsupervised learning algorithm for dealing with online English-Chinese name transliteration. Our experimental results show that our proposed new methods can make great improvements compared with the previous search-result-based term translation extraction methods. Keywords: CLIR, Transliteration, Unknown Term Translation, Web Search Result, Machine Translation. ∗ Dept. of Computer Sci. and Eng., National Cheng Kung University, No.1, University Road, Tainan City 701, Taiwan (R.O.C.) Tel.: +886-6-2757575 ext: 62545 Fax: +886-6-2747076 E-mail: whlu@mial.ncku.edu.tw, ys.chang1976@gmail.com + Penpower Technology Ltd. 7F, NO.47, Lane 2, Sec. 2, Guanfu Rd., Hsinchu City 300, Taiwan, R.O.C. Tel: +886-3-5722691 Fax: +886-3-5716243 E-mail: hunter@penpower.com.tw 92 Wen-Hsiang Lu et al. 1. Introduction Many existing cross-language information retrieval (CLIR) systems [Ballesteros and Croft 1997; Hull and Grefenstette 1996] encounter great difficulties in dealing with unknown term translation since these systems rely mostly on general-purpose bilingual dictionaries, which usually lack translations of abbreviations and proper names. Moreover, according to the report in a previous work [Cheng et al. 2004], even for frequent Web queries, about 64% of them are not covered in an English-Chinese lexicon with about 120K entries (provided by Linguistic Data Consortium). However, several automatic translation extraction methods based on parallel [Brown et al. 1993; Melamed 2000; Nie et al. 1999; Smadja et al. 1996] or comparable corpora [Rapp 1999; Fung and Yee 1998] eventually suffer from the problems of insufficient parallel texts and the shortage of translation accuracy of comparable corpora in various subject domains. The Web has been expanded with an enormous amount of multilingual hypertext resources in diverse subjects. Recently, a number of studies in natural language processing (NLP) have concentrated on the use of Web resources to complement insufficient text corpora [Cao and Li 2002; Kilgarriff and Grefenstette 2003]. To automatically collect huge amounts of parallel corpora from the Web in various domains, some researchers have developed feasible techniques of utilizing similar file names, text length, and …

Content truncated to 500 words