Dragonfly: Advances in Non-Speaker Annotation for Low Resource Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6983–6987 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6983 Dragonfly: Advances in Non-Speaker Annotation for Low Resource Languages Cash Costello, Shelby Anderson, Caitlyn Bishop, James Mayfield, Paul McNamee Johns Hopkins University Applied Physics Laboratory 11100 Johns Hopkins Road, Laurel, MD 20723 USA {cash.costello, shelby.anderson, caitlyn.bishop, james.mayfield, paul.mcnamee}@jhuapl.edu Abstract Dragonfly is an open source software tool that supports annotation of text in a low resource language by non-speakers of the language. Using semantic and contextual information, non-speakers of a language familiar with the Latin script can produce high quality named entity annotations to support construction of a name tagger. We describe a procedure for annotating low resource languages using Dragonfly that others can use, which we developed based on our experience annotating data in more than ten languages. We also present performance comparisons between models trained on native speaker and non-speaker annotations. Keywords: named entity recognition, non-speaker annotation, low resource languages 1. Introduction Neural named entity recognition (NER) models require large amounts of training data to achieve state-of-the-art performance. A significant challenge with low resource languages is locating native speakers to perform these an- notation tasks. To address this, we have created a tool for annotating documents in a language that the annotators do not speak. Our tool, Dragonfly, has been improved through our experiences annotating text in 13 different languages. It is available as open source software on GitHub.1 The development of this tool was motivated by the LoReHLT evaluations (Christianson et al., 2018), which included an evaluation of NER for surprise low resource languages with tight time constraints (approximately 1 to 2 weeks depending on the year). Three of the teams adopted non-speaker annotation as a key enabling technology for the evaluation, and developed annotation tools in paral- lel. TALEN (Mayhew and Roth, 2018) transforms the text towards English by transliterating non-Latin scripts and performing word replacement using a bilingual dictionary. The ELISA IE annotation tool is described in (Lin et al., 2018) along with an early version of Dragonfly. Both of these tools display the original text with transliterations and translations shown at the sentence level. This paper highlights the advances we have made to Dragonfly based on two additional evaluations. These advances make non- speaker annotation both faster and more accurate. For each low resource language in the evaluation, monolin- gual text was provided from multiple sources (news, dis- cussion forum/blogs, and Twitter). In addition, a bilingual lexicon, a grammar sketch, and a small amount of paral- lel text were included in the language pack. The minimum set of resources required for annotating with Dragonfly is the monolingual text in the language of interest and a small bilingual lexicon with some names of entities in it. 2. Requirements When annotating the names of entities in an unfamiliar lan- guage, the annotator comes across two types of entities. The first of these are entities that are already known to …

Content truncated to 500 words