Introducing NIEUW: Novel Incentives and Workflows for Eliciting Linguistic Data

Introducing NIEUW: Novel Incentives and Workflows for Eliciting Linguistic Data Christopher Cieri, James Fiumara, Mark Liberman, Chris Callison-Burch*, Jonathan Wright University of Pennsylvania Linguistic Data Consortium and *Department of Computer and Information Science {ccieri, jfiumara, myl, jdwright} @ldc.upenn.edu, {ccb} @cis.upenn.edu Abstract This paper introduces the NIEUW (Novel Incentives and Workflows) project funded by the United States National Science Foundation and part of the Linguistic Data Consortium’s strategy to provide order of magnitude improvement in the scale, cost, variety, linguistic diversity and quality of Language Resources available for education, research and technology development. Notwithstanding decades of effort and progress in collecting and distributing Language Resources, it remains the case that demand still far exceeds supply for all of the approximately 7000 languages in the world, even the most well documented languages with global economic and political influence. The absence of Language Resources, regardless of the language, stifles teaching and technology building, inhibiting the creation of language enabled applications and, as a result, commerce and communication. Project oriented approaches which focus intensive funding and effort on problems of limited scope over short durations can only address part of the problem. The HLT community instead requires approaches that do not rely upon highly constrained resources such as project funding and can be sustained across many languages and many years. In this paper, we describe a new initiative to harness the power of alternative incentives to elicit linguistic data and annotation. We also describe changes to the workflows necessary to collect data from workforces attracted by these incentives. Keywords: novel incentives, workflows, language resources 1. Introduction & Motivation The Human Language Technology (HLT) community has benefitted from massive contributions of linguistic data from data centers, governments and groups around the world. Nevertheless, potential still remains largely untapped because the LRs that fuel development fall far short of need whether measured by volume, data type, or language coverage. Data centers regularly receive requests for data sets that they cannot supply even for the dozen languages with the greatest populations and gross linguistic product. A 2010 survey (METANET) of the Language Resources (LRs) required to build the HLTs that would support speakers of a given language working in the information age found that none of the language of the European Union, not even English, were fully supplied and warned that “21 out of 30 European languages could become extinct in the digital world”. Beyond a few exceptional cases: Mandarin, Modern Standard Arabic, Japanese, language spoken primarily outside the EU, suffer even greater LR deficits. Although the absence of language resources hinders the development of technologies to support international commerce, the problem becomes truly acute during emergent situations such as natural disasters and refugee crises. The International Association of Conference Interpreters warned in 2008: “Ending a conflict and delivering emergency and humanitarian aid across language barriers represents a major challenge, for which few of the organisations entrusted with operations in the field are well equipped. This problem is compounded by the fact that there is a chronic …

Content truncated to 500 words