The Human Language Project: Building a Universal Corpus of the World’s Languages

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 88–97, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics The Human Language Project: Building a Universal Corpus of the World’s Languages Steven Abney University of Michigan abney@umich.edu Steven Bird University of Melbourne and University of Pennsylvania sbird@unimelb.edu.au Abstract We present a grand challenge to build a corpus that will include all of the world’s languages, in a consistent structure that permits large-scale cross-linguistic pro- cessing, enabling the study of universal linguistics. The focal data types, bilin- gual texts and lexicons, relate each lan- guage to one of a set of reference lan- guages. We propose that the ability to train systems to translate into and out of a given language be the yardstick for determin- ing when we have successfully captured a language. We call on the computational linguistics community to begin work on this Universal Corpus, pursuing the many strands of activity described here, as their contribution to the global effort to docu- ment the world’s linguistic heritage before more languages fall silent. 1 Introduction The grand aim of linguistics is the construction of a universal theory of human language. To a com- putational linguist, it seems obvious that the first step is to collect significant amounts of primary data for a large variety of languages. Ideally, we would like a complete digitization of every human language: a Universal Corpus. If we are ever to construct such a corpus, it must be now. With the current rate of language loss, we have only a small window of opportunity before the data is gone forever. Linguistics may be unique among the sciences in the crisis it faces. The next generation will forgive us for the most egregious shortcomings in theory construction and technol- ogy development, but they will not forgive us if we fail to preserve vanishing primary language data in a form that enables future research. The scope of the task is enormous. At present, we have non-negligible quantities of machine- readable data for only about 20–30 of the world’s 6,900 languages (Maxwell and Hughes, 2006). Linguistics as a field is awake to the crisis. There has been a tremendous upsurge of interest in doc- umentary linguistics, the field concerned with the the “creation, annotation, preservation, and dis- semination of transparent records of a language” (Woodbury, 2010). However, documentary lin- guistics alone is not equal to the task. For example, no million-word machine-readable corpus exists for any endangered language, even though such a quantity would be necessary for wide-ranging in- vestigation of the language once no speakers are available. The chances of constructing large-scale resources will be greatly improved if computa- tional linguists contribute their expertise. This collaboration between linguists and com- putational linguists will extend beyond the con- struction of the Universal Corpus to its exploita- tion for both theoretical and technological ends. We envisage a new paradigm of universal linguis- tics, in which grammars of individual languages are built from the ground …

Content truncated to 500 words