Named Entity Recognition for Linguistic Rapid Response in Low-Resource Languages: Sorani Kurdish and Tajik

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 998–1006, Osaka, Japan, December 11-17 2016. Named Entity Recognition for Linguistic Rapid Response in Low-Resource Languages: Sorani Kurdish and Tajik Patrick Littell, Kartik Goyal, David Mortensen, Alexa Little, Chris Dyer, Lori Levin Carnegie Mellon University Language Technologies Institute 5000 Forbes Ave., Pittsburgh PA 15213 plittell@cs.cmu.edu, kartikgo@cs.cmu.edu, dmortens@cs.cmu.edu alexanicolelittle@gmail.com, cdyer@cs.cmu.edu, lsl@cs.cmu.edu Abstract This paper describes our construction of named-entity recognition (NER) systems in two West- ern Iranian languages, Sorani Kurdish and Tajik, as a part of a pilot study of Linguistic Rapid Response to potential emergency humanitarian relief situations. In the absence of large annotated corpora, parallel corpora, treebanks, bilingual lexica, etc., we found the following to be effective: exploiting distributional regularities in monolingual data, projecting information across closely related languages, and utilizing human linguist judgments. We show promising results on both a four-month exercise in Sorani and a two-day exercise in Tajik, achieved with minimal annotation costs. 1 Introduction This paper describes our rapid construction of NER systems, as a part of a pilot study of Linguistic Rapid Response to potential emergency humanitarian relief situations. When a disaster strikes a community that speaks a low-resource language without an existing NLP infrastructure, how long would it take to put such an infrastructure, however imperfect, in place? What kinds of systems can be in place within 24 or 48 hours, and what levels of performance can we expect? What resources – besides existing gazetteers, parallel corpora, etc. – can be assembled in this timeframe and brought to bear on NER? Contemporary techniques for creating natural language processing (NLP) tools are dominated by su- pervised learning approaches, in which large quantities of high quality data are annotated in a task-specific fashion and utilized along with manually-built collaborate resources like WordNet, Ontonotes, etc. These techniques have provided very good performance, but with 7,000 languages in the world these resources cannot feasibly be compiled in advance, and could not be assembled within an emergency timeframe. Following many examples of “surprise language exercises” (Oard, 2003), the work described in this paper tackles the problem of rapid development of important NLP tools and resources, with a focus on NER, for low resource languages. We assume that for a “low resource” language, there is some monolingual corpus data and little to no annotated data for supervised training. A major theme underlying this work is a focus on building “omnivorous” models and pipelines that, instead of relying on elaborate and robust linguistic resources compiled ahead of time, try to opportunistically incorporate linguistic theory, informal and non-expert intuitions about the language and task at hand, and resources adapted from closely related languages, all while avoiding extensive manual annotation. An important feature of our work is the use of human linguists who do not speak the language but are familiar with the structure of human languages in general, have some knowledge about the language family in question, and can absorb facts about the language quickly by reading …

Content truncated to 500 words