Enhancement of Lexical Concepts Using Cross-lingual Web Mining

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 852–861, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Enhancement of Lexical Concepts Using Cross-lingual Web Mining Dmitry Davidov ICNC The Hebrew University of Jerusalem dmitry@alice.nc.huji.ac.il Ari Rappoport Institute of Computer Science The Hebrew University of Jerusalem arir@cs.huji.ac.il Abstract Sets of lexical items sharing a significant aspect of their meaning (concepts) are fun- damental in linguistics and NLP. Manual concept compilation is labor intensive, er- ror prone and subjective. We present a web-based concept extension algorithm. Given a set of terms specifying a concept in some language, we translate them to a wide range of intermediate languages, disambiguate the translations using web counts, and discover additional concept terms using symmetric patterns. We then translate the discovered terms back into the original language, score them, and ex- tend the original concept by adding back- translations having high scores. We eval- uate our method in 3 source languages and 45 intermediate languages, using both hu- man judgments and WordNet. In all cases, our cross-lingual algorithm significantly improves high quality concept extension. 1 Introduction A concept (or lexical category) is a set of lex- ical items sharing a significant aspect of their meanings (e.g., types of food, tool names, etc). Concepts are fundamental in linguistics and NLP, in thesauri, dictionaries, and various applications such as textual entailment and question answering. Great efforts have been invested in manual preparation of concept resources such as WordNet (WN). However, manual preparation is labor in- tensive, which means it is both costly and slow to update. Applications needing data on some very specific domain or on a recent news-related event may find such resources lacking. In addition, manual preparation is error-prone and susceptible to subjective concept membership decisions, fre- quently resulting in concepts whose terms do not belong to the same level of granularity1. As a re- sult, there is a need to find methods for automatic improvement of concept coverage and quality. The web is a huge up-to-date corpus covering many domains, so using it for concept extension has the potential to address the above problems. The majority of web pages are written in a few salient languages, hence most of the web-based in- formation retrieval studies are done on these lan- guages. However, due to the substantial growth of the multilingual web2, languages in which concept terms are expressed in the most precise manner frequently do not match the language where in- formation is needed. Moreover, representations of the same concept in different languages may com- plement each other. In order to benefit from such cross-lingual in- formation, concept acquisition systems should be able to gather concept terms from many available languages and convert them to the desired lan- guage. In this paper we present such an algorithm. Given a set of words specifying a concept in some source language, we translate them to a range of intermediate languages and disambiguate the translations using web counts. Then we discover additional concept terms …

Content truncated to 500 words