Cross-lingual Character-Level Neural Morphological Tagging

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 748–759 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Cross-lingual, Character-Level Neural Morphological Tagging Ryan Cotterell@ and Georg HeigoldK @Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA KGerman Research Center for Artificial Intelligence, Saarbr¨ucken, SL, GER ryan.cotterell@jhu.edu georg.heigold@gmail.com Abstract Even for common NLP tasks, sufficient supervision is not available in many languages—morphological tagging is no exception. In the work presented here, we explore a transfer learning scheme, whereby we train character-level recurrent neural taggers to predict morphological taggings for high-resource languages and low-resource languages together. Learning joint character representations among mul- tiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones, im- proving accuracy by up to 30%. 1 Introduction State-of-the-art morphological taggers require thou- sands of annotated sentences to train. For the ma- jority of the world’s languages, however, sufficient large-scale annotation is not available and obtain- ing it would often be infeasible. Accordingly, an important road forward in low-resource NLP is the development of methods that allow for the training of high-quality tools from smaller amounts of data. In this work, we focus on transfer learning—we train a recurrent neural tagger for a low-resource language jointly with a tagger for a related high- resource language. Forcing the models to share character-level features among the languages al- lows large gains in accuracy when tagging the low- resource languages, while maintaining (or even im- proving) accuracy on the high-resource language. Recurrent neural networks constitute the state of the art for a myriad of tasks in NLP, e.g., multi- lingual part-of-speech tagging (Plank et al., 2016), syntactic parsing (Dyer et al., 2015; Zeman et al., 2017), morphological paradigm completion (Cot- terell et al., 2016, 2017) and language modeling (Sundermeyer et al., 2012; Melis et al., 2017); re- cently, such models have also improved morpho- logical tagging (Heigold et al., 2016, 2017). In addition to increased performance over classical approaches, neural networks also offer a second advantage: they admit a clean paradigm for multi- task learning. If the learned representations for all of the tasks are embedded jointly into a shared vector space, the various tasks reap benefits from each other and often performance improves for all (Collobert et al., 2011b). We exploit this idea for language-to-language transfer to develop an approach for cross-lingual morphological tagging. We experiment on 18 languages taken from four different language families. Using the Univer- sal Dependencies treebanks, we emulate a low- resource setting for our experiments, e.g., we at- tempt to train a morphological tagger for Catalan using primarily data from a related language like Spanish. Our results demonstrate the successful transfer of morphological knowledge from the high- resource languages to the low-resource languages without relying on an externally acquired bilingual lexicon or bitext. We consider both the single- and multi-source transfer case and explore how similar two languages must be in order to enable high- quality transfer of morphological taggers.1 2 …

Content truncated to 500 words