Inducing a Multilingual Dictionary from a Parallel Multitext in Related Languages

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 875–882, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Inducing a multilingual dictionary from a parallel multitext in related languages Dmitriy Genzel Department of Computer Science Box 1910 Brown University Providence, RI 02912, USA dg@cs.brown.edu Abstract Dictionaries and word translation models are used by a variety of systems, espe- cially in machine translation. We build a multilingual dictionary induction system for a family of related resource-poor lan- guages. We assume only the presence of a single medium-length multitext (the Bible). The techniques rely upon lexical and syntactic similarity of languages as well as on the fact that building dictionar- ies for several pairs of languages provides information about other pairs. 1 Introduction and Motivation Modern statistical natural language processing tech- niques require large amounts of human-annotated data to work well. For practical reasons, the required amount of data exists only for a few languages of major interest, either commercial or governmental. As a result, many languages have very little com- putational research done in them, especially outside the borders of the countries in which these languages are spoken. Some of these languages are, however, major languages with hundreds of millions of speak- ers. Of the top 10 most spoken languages, Lin- guistic Data Consortium at University of Pennsyl- vania, the premier U.S. provider of corpora, offers text corpora only in 7 (The World Factbook (2004), 2000 estimate) Only a few of the other languages (French, Arabic, and Czech) have resources pro- vided by LDC. Many Asian and Eastern European languages number tens of millions of speakers, yet very few of these seem to have any related compu- tational linguistics work, at least as presented at the international conferences, such as the ACL.1 The situation is not surprising, nor is it likely to significantly change in the future. Luckily, most of these less-represented languages belong to lan- guage families with several prominent members. As a result, some of these languages have siblings with more resources and published research. 2 Inter- estingly, the better-endowed siblings are not always the ones with more native speakers, since political considerations are often more important.3 If one is able to use the resources available in one lan- guage (henceforth referred to as source) to facilitate the creation of tools and resource in another, related language (target), this problem would be alleviated. This is the ultimate goal of this project, but in the first stage we focus on multi-language dictionary in- duction. Building a high-quality dictionary, or even bet- ter, a joint word distribution model over all the lan- guages in a given family is very important, because using such a model one can use a variety of tech- niques to project information across languages, e.g. to parse or to translate. Building a unified model for more than a pair of languages improves the quality over building several unrelated pairwise models, be- cause relating them to each other provides additional …

Content truncated to 500 words