Morphological Segmentation for Low Resource Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3996–4002 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3996 Morphological Segmentation for Low Resource Languages Justin Mott1, Ann Bies1, Stephanie Strassel1, Jordan Kodner2, Caitlin Richter2, Hongzhi Xu2,3, Mitch Marcus2 1Linguistic Data Consortium, University of Pennsylvania 3600 Market Street, Suite 810, Philadelphia, PA 19104 2Department of Computer and Information Science, University of Pennsylvania 3330 Walnut Street, Philadelphia, PA 19104 3ICSA, Shanghai International Studies University 1550 Wenxiang Road, Shanghai 201600, China {jmott, bies, strassel}@ldc.upenn.edu, {jkodner, ricca}@sas.upenn.edu, {xh, mitch}@cis.upenn.edu Abstract This paper describes a new morphology resource created by Linguistic Data Consortium and the University of Pennsylvania for the DARPA LORELEI Program. The data consists of approximately 2000 tokens annotated for morphological segmentation in each of 9 low resource languages, along with root information for 7 of the languages. The languages annotated show a broad diversity of typological features. A minimal annotation scheme for segmentation was developed such that it could capture the patterns of a wide range of languages and also be performed reliably by non-linguist annotators. The basic annotation guidelines were designed to be language-independent, but included language-specific morphological paradigms and other specifications. The resulting annotated corpus is designed to support and stimulate the development of unsupervised morphological segmenters and analyzers by providing a gold standard for their evaluation on a more typologically diverse set of languages than has previously been available. By providing root annotation, this corpus is also a step toward supporting research in identifying richer morphological structures than simple morpheme boundaries. Keywords: low resource languages, morphology, morphological analyzers, morphological segmenters, annotation, data centers 1. Introduction and Motivation This paper describes a new morphology resource created by Linguistic Data Consortium and the University of Pennsylvania for the DARPA LORELEI Program. The data consists of approximately 2000 tokens annotated for morphological segmentation for each of 9 low resource languages, along with root information for 7 of the languages. The languages annotated show a broad diversity of typological features. This annotated data will support the development of unsupervised morphological segmenters and analyzers by providing a gold standard for their evaluation. In the sections that follow we discuss the goals of the LORELEI program and the morphological segmentation task, related work, the annotation procedures and resulting corpus, research results and future directions. The data described here is included in the LORELEI Representative Language Packs scheduled for publication in the LDC Catalog starting in 2020. 1.1 The LORELEI Program The DARPA Low Resource Languages for Emerging Incidents (LORELEI) Program aims to advance the capabilities of human language technologies in low- resource languages, with a particular focus on rapidly obtaining situational awareness after the emergence of an unexpected incident such as a natural disaster in a setting where technology does not yet exist for the local language (DARPA, 2014). Linguistic Data Consortium (LDC) has built a variety of linguistic resources for nearly three dozen low resource languages for the LORELEI program. Representative Language Packs – …

Content truncated to 500 words