On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 216–226 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 216 On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages Yi Zhu1∗, Benjamin Heinzerling2,3∗, Ivan Vuli´c1, Michael Strube4, Roi Reichart5, Anna Korhonen1 1Language Technology Lab, University of Cambridge 2RIKEN AIP, 3Tohoku University 4Heidelberg Institute for Theoretical Studies, 5Technion, IIT {yz568,iv250,alk23}@cam.ac.uk, benjamin.heinzerling@riken.jp michael.strube@h-its.org, roiri@technion.ac.il Abstract Recent work has validated the importance of subword information for word representation learning. Since subwords increase parameter sharing ability in neural models, their value should be even more pronounced in low-data regimes. In this work, we therefore provide a comprehensive analysis focused on the use- fulness of subwords for word representation learning in truly low-resource scenarios and for three representative morphological tasks: fine-grained entity typing, morphological tag- ging, and named entity recognition. We con- duct a systematic study that spans several dimensions of comparison: 1) type of data scarcity which can stem from the lack of task- specific training data, or even from the lack of unannotated data required to train word em- beddings, or both; 2) language type by work- ing with a sample of 16 typologically diverse languages including some truly low-resource ones (e.g. Rusyn, Buryat, and Zulu); 3) the choice of the subword-informed word repre- sentation method. Our main results show that subword-informed models are universally use- ful across all language types, with large gains over subword-agnostic embeddings. They also suggest that the effective use of subwords largely depends on the language (type) and the task at hand, as well as on the amount of avail- able data for training the embeddings and task- based models, where having sufficient in-task data is a more critical requirement. 1 Introduction and Motivation Recent studies have confirmed the usefulness of leveraging subword-level information in learning word representations (Peters et al., 2018; Heinz- erling and Strube, 2018; Grave et al., 2018; Zhu et al., 2019, inter alia), and in a range of tasks such as sequence tagging (Lample et al., 2016; Akbik ∗Equal contribution, work partly done while at HITS. et al., 2018; Devlin et al., 2019), fine-grained entity typing (Zhu et al., 2019), neural machine transla- tion (Sennrich et al., 2016; Luong and Manning, 2016; Lample et al., 2018; Durrani et al., 2019), or general and rare word similarity (Pilehvar et al., 2018; Zhu et al., 2019). The subword-informed word representation architectures leverage the in- ternal structure of words and assume that a word’s meaning can be inferred from the meaning of its constituent (i.e., subword) parts. Instead of treating each word as an atomic unit, subword-informed neural architectures reduce data sparsity by relying on parameterization at the level of subwords (Bo- janowski et al., 2017; Pinter et al., 2017; Chaudhary et al., 2018; Kudo, 2018). An increasing body of work focuses on various aspects of subword-informed representation learn- ing such as segmentation of words into subwords and composing subword embeddings into word rep- resentations (Lazaridou et al., 2013; …

Content truncated to 500 words