A Systematic Study of Leveraging Subword Information for Learning Word Representations

Proceedings of NAACL-HLT 2019, pages 912–932 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 912 A Systematic Study of Leveraging Subword Information for Learning Word Representations Yi Zhu, Ivan Vuli´c, and Anna Korhonen Language Technology Lab University of Cambridge {yz568, iv250, alk23}@cam.ac.uk Abstract The use of subword-level information (e.g., characters, character n-grams, morphemes) has become ubiquitous in modern word rep- resentation learning. Its importance is attested especially for morphologically rich languages which generate a large number of rare words. Despite a steadily increasing interest in such subword-informed word representations, their systematic comparative analysis across typo- logically diverse languages and different tasks is still missing. In this work, we deliver such a study focusing on the variation of two cru- cial components required for subword-level in- tegration into word representation models: 1) segmentation of words into subword units, and 2) subword composition functions to obtain fi- nal word representations. We propose a gen- eral framework for learning subword-informed word representations that allows for easy ex- perimentation with different segmentation and composition components, also including more advanced techniques based on position em- beddings and self-attention. Using the unified framework, we run experiments over a large number of subword-informed word representa- tion configurations (60 in total) on 3 tasks (gen- eral and rare word similarity, dependency pars- ing, fine-grained entity typing) for 5 languages representing 3 language types. Our main re- sults clearly indicate that there is no “one-size- fits-all” configuration, as performance is both language- and task-dependent. We also show that configurations based on unsupervised seg- mentation (e.g., BPE, Morfessor) are some- times comparable to or even outperform the ones based on supervised word segmentation. 1 Introduction Word representations are central to a wide vari- ety of NLP tasks (Collobert et al., 2011; Chen and Manning, 2014; Jia and Liang, 2016; Ammar et al., 2016; Goldberg, 2017; Peters et al., 2018; Kudo, 2018, inter alia). Standard word representa- tion models are based on the distributional hypoth- esis (Harris, 1954) and induce representations from large unlabeled corpora using word co-occurrence statistics (Mikolov et al., 2013; Pennington et al., 2014; Levy and Goldberg, 2014). However, as pointed out by recent work (Bojanowski et al., 2017; Vania and Lopez, 2017; Pinter et al., 2017; Chaudhary et al., 2018; Zhao et al., 2018), mapping a finite set of word types into corresponding word representations limits the capacity of these models to learn beyond distributional information, which leads to several fundamental limitations. The standard approaches ignore the internal structure of words, that is, the syntactic or seman- tic composition from subwords or morphemes to words, and are incapable of parameter sharing at the level of subword units. Assigning only a sin- gle vector to each word causes the data sparsity problem, especially in resource-poor settings where huge amounts of training data cannot be guaran- teed. The issue is also prominent for morphologi- cally rich languages (e.g., Finnish) with productive morphological systems that generate a large num- ber of infrequent/rare words (Gerz et …

Content truncated to 500 words