Constraint-Based Bilingual Lexicon Induction for Closely Related Languages

Constraint-Based Bilingual Lexicon Induction for Closely Related Languages Arbi Haza Nasution, Yohei Murakami, Toru Ishida Department of Social Informatics, Kyoto University Yoshida-Honmachi, Sakyo-Ku, Kyoto, 606-8501, Japan arbi@ai.soc.i.kyoto-u.ac.jp, {yohei,ishida}@i.kyoto-u.ac.jp Abstract The lack or absence of parallel and comparable corpora makes bilingual lexicon extraction becomes a difficult task for low-resource languages. Pivot language and cognate recognition approach have been proven useful to induce bilingual lexicons for such languages. We analyze the features of closely related languages and define a semantic constraint assumption. Based on the assumption, we propose a constraint-based bilingual lexicon induction for closely related languages by extending constraints and translation pair candidates from recent pivot language approach. We further define three constraint sets based on language characteristics. In this paper, two controlled experiments are conducted. The former involves four closely related language pairs with different language pair similarities, and the latter focuses on sense connectivity between non-pivot words and pivot words. We evaluate our result with F-measure. The result indicates that our method works better on voluminous input dictionaries and high similarity languages. Finally, we introduce a strategy to use proper constraint sets for different goals and language characteristics. Keywords: bilingual lexicon induction, constraint satisfaction, Weighted Partial MaxSAT 1. Introduction Machine readable bilingual dictionary is very useful in information retrieval and natural language processing re- searches, yet usually unavailable for low resource lan- guages. Previous work shows the effectiveness of paral- lel corpora (Fung, 1998) and comparable corpora (Li and Gaussier, 2010) in inducing bilingual lexicon for high- resource languages. Bilingual lexicons extraction becomes a difficult task for low-resource languages due to the lack or absence of parallel and comparable corpora. Pivot lan- guage (Tanaka and Umemura, 1994) and cognate recog- nition (Mann and Yarowsky, 2001) approaches have been proven useful to induce bilingual lexicon for low-resource languages. Recently, our team (Wushouer et al., 2015) showed a promising approach of treating pivot-based bilin- gual lexicon induction for low-resource languages as opti- mization problem. Their method is based on the assump- tion that lexicons of closely related languages offer one-to- one mapping. However, this one-to-one assumption is too strong to suit languages which have the one-to-many trans- lation characteristic. Therefore, we aim at extending the constraint-based bilingual lexicon induction to support any other closely related languages. To this end, we address the following two research goals: • Generalize constraint-based bilingual lexicon induc- tion framework: We extend the one-to-one approach constraints and translation pair candidates to get one- to-many translation results, and further generalize the framework to cover any closely related languages with either one-to-one or one-to-many translation charac- teristics. • Identify the best constraint set according to lan- guage pairs: The similarity between closely related languages varies based on their genetic relationship. Languages are classified into low-resource and high- resource languages based on their number/amount of resources, including size of dictionaries. Therefore, we identify characteristics of language that affect the performance and further identify the best constraint set for different language characteristics. Closely related languages with high similarity between the input …

Content truncated to 500 words