LORELEI Language Packs: Data, Tools, and Resources for Technology Development in Low Resource Languages

LORELEI Language Packs: Data, Tools, and Resources for Technology Development in Low Resource Languages Stephanie Strassel and Jennifer Tracey Linguistic Data Consortium 3600 Market Street, Suite 810 Philadelphia, PA 19104 E-mail: strassel@ldc.upenn.edu, garjen@ldc.upenn.edu Abstract In this paper, we describe the textual linguistic resources in nearly 3 dozen languages being produced by Linguistic Data Consortium for DARPA’s LORELEI (Low Resource Languages for Emergent Incidents) Program. The goal of LORELEI is to improve the performance of human language technologies for low-resource languages and enable rapid re-training of such technologies for new languages, with a focus on the use case of deployment of resources in sudden emergencies such as natural disasters. Representative languages have been selected to provide broad typological coverage for training, and surprise incident languages for testing will be selected over the course of the program. Our approach treats the full set of language packs as a coherent whole, maintaining LORELEI-wide specifications, tag sets and guidelines, while allowing for adaptation to the specific needs created by each language. Each representative language corpus, therefore, both stands on its own as a resource for the specific language and forms part of a large multilingual resource for broader cross-language technology development. Keywords: low resource languages, multilingual resources, situational awareness 1. Introduction The goal of DARPA’s LORELEI (Low Resource Languages for Emergent Incidents) Program is to improve the performance of human language technologies for low-resource languages, particularly in the context of a rapidly emerging and quickly evolving situation like a natural disaster or disease outbreak. LORELEI systems will be required to process information about topics, entities, events and sentiment found in the LORELEI data, with the goal of providing situational awareness within days or even hours of the outbreak of an incident. Linguistic Data Consortium is building text language packs for LORELEI 1 , comprising data, annotations, natural language processing tools, lexicons and grammatical resources for 23 Representative Languages as well as 12 Incident Languages, listed in Table 1. Representative Languages (RL) have been selected to provide broad typological coverage, while Incident Languages (IL) have been selected to enable development and testing of LORELEI system capabilities. The choice of evaluation Incident Languages remains unknown to performers until the start of the evaluation. There is a growing interest in human language technology for low resource languages (LRLs). The IARPA Babel program targets improvements in Automatic Speech Recognition (ASR) and Keyword Spotting (KWS) system performance, and language packs for Babel primarily consist of transcribed and untranscribed speech in a variety of acoustic conditions (IARPA 2016). While Babel focuses on general improvements to speech processing technology for 1 A small amount of speech data will also be created for each LORELEI language, under a separate effort by Appen. LRLs, LORELEI language packs (and the corresponding technology evaluations) are designed specifically with the goal of improved technology for situational awareness in emergent situations, and their composition reflects this. Table 1: LORELEI Program Languages This focus also provides a contrast between LORELEI and LDC’s earlier work building language packs for …

Content truncated to 500 words