Cross Language Text Categorization Using a Bilingual Lexicon

Cross Language Text Categorization Using a Bilingual Lexicon Ke Wu, Xiaolin Wang and Bao-Liang Lu∗ Department of Computer Science and Engineering, Shanghai Jiao Tong University 800 Dong Chuan Rd., Shanghai 200240, China {wuke,arthur general,bllu}@sjtu.edu.cn Abstract With the popularity of the Internet at a phe- nomenal rate, an ever-increasing number of documents in languages other than English are available in the Internet. Cross lan- guage text categorization has attracted more and more attention for the organization of these heterogeneous document collections. In this paper, we focus on how to con- duct effective cross language text catego- rization. To this end, we propose a cross language naive Bayes algorithm. The pre- liminary experiments on collected document collections show the effectiveness of the pro- posed method and verify the feasibility of achieving performance close to monolingual text categorization, using a bilingual lexicon alone. Also, our algorithm is more efficient than our baselines. 1 Introduction Due to the popularity of the Internet, an ever- increasing number of documents in languages other than English are available in the Internet. The or- ganization of these heterogeneous document collec- tions increases cost of human labor significantly. On the one hand, experts who know different languages are required to organize these collections. On the other hand, maybe there exist a large amount of la- belled documents in a language (e.g. English) which are in the same class structure as the unlabelled doc- uments in another language. As a result, how to ex- ∗Corresponding author. ploit the existing labelled documents in some lan- guage (e.g. English) to classify the unlabelled doc- uments other than the language in multilingual sce- nario has attracted more and more attention (Bel et al., 2003; Rigutini et al., 2005; Olsson et al., 2005; Fortuna and Shawe-Taylor, 2005; Li and Shawe- Taylor, 2006; Gliozzo and Strapparava, 2006). We refer to this task as cross language text categoriza- tion. It aims to extend the existing automated text categorization system from one language to other languages without additional intervention of human experts. Formally, given two document collections {De, Df} from two different languages e and f re- spectively, we use the labelled document collection De in the language e to deduce the labels of the doc- ument collection Df in the language f via an algo- rithm A and some external bilingual resources. Typically, some external bilingual lexical re- sources, such as machine translation system (MT), large-scale parallel corpora and multilingual ontol- ogy etc., are used to alleviate cross language text categorization. However, it is hard to obtain them for many language pairs. In this paper, we focus on using a cheap bilingual resource, e.g. bilingual lexi- con without any translation information, to conduct cross language text categorization. To my knowl- edge, there is little research on using a bilingual lex- icon alone for cross language text categorization. In this paper, we propose a novel approach for cross language text categorization via a bilingual lexicon alone. We call this approach as Cross Lan- guage Naive Bayes Classifier …

Content truncated to 500 words