Selection Criteria for Low Resource Language Programs

Selection Criteria for Low Resource Language Programs Christopher Cieri○, Mike Maxwell◻, Stephanie Strassel○, Jennifer Tracey○ ○ Linguistic Data Consortium. University of Pennsylvania 3600 Market Street, Suite 810, Philadelphia, PA. 19104 USA email AT ︎ldc.upenn.edu ◻ University of Maryland College Park, MD 20742, USA email AT umd.edu {ccieri, mmaxwell, strassel, garjen} Abstract This paper documents and describes the criteria used to select languages for study within programs that include low resource languages whether given that label or another similar one. It focuses on five US common task, Human Language Technology research and development programs in which the authors have provided information or consulting related to the choice of language. The paper does not describe the actual selection process which is the responsibility of program management and highly specific to a program’s individual goals and context. Instead it concentrates on the data and criteria that have been considered relevant previously with the thought that future program managers and their consultants may adapt these and apply them with different prioritization to future programs. Keywords: language resources, low resource languages, common task programs 1. Introduction The past 10 years have seen significant growth in work on resource-poor languages within the Human Language Technology (HLT) research community. Whether one sees this growth as the natural outcome of successful HLT development in well-resourced languages or as an opportunity to test the generality of HLT, the shift in focus is undeniable. Within the United States alone, the TIDES, REFLEX LCTL, Babel and LORELEI programs have all focused on developing language resources and technologies for low resource languages. However, differences in the terminology, available information and goals of low resource language efforts lead to variability and some obscurity in the language selection process. Moving beyond the four US programs named above, there is an even greater range of motives for studying low resources languages. For example, while the LORELEI program, described in greater detail below, seeks technologies to facilitate situational awareness in the event of a disaster, the EU funded METANET (2010) program asserts that “The majority of European languages are severely under-resourced” and proposes that a “coordinated, large-scale effort has to be made in Europe to create the missing technologies and transfer this technology to the languages faced with digital extinction”. The motivations for the proposed effort include quality of life, information access and the ability to collaborate across multilingual Europe. The US National Science Foundation’s Documenting Endangered Languages program (2014) gives very different motivations: “Most of what is known about human communication and cognition is based on less than 10 percent of the world's 7,000 languages. We must do our best to document living endangered languages and their associated cultural and scientific information before they disappear.” Such differences in motivation clearly lead to very different languages studied and differences in the languages studied affect opportunities for collaboration across programs. In addition to the programs’ commitments of time and finances, the new language resources (LRs) they create are critical for bringing HLTs to new languages, a matter …

Content truncated to 500 words