From Characters to Words to in Between: Do We Capture Morphology?

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2016–2027 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1184 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2016–2027 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1184 From Characters to Words to in Between: Do We Capture Morphology? Clara Vania and Adam Lopez Institute for Language, Cognition and Computation School of Informatics University of Edinburgh c.vania@ed.ac.uk, alopez@inf.ed.ac.uk Abstract Words can be represented by composing the representations of subword units such as word segments, characters, and/or char- acter n-grams. While such representations are effective and may capture the mor- phological regularities of words, they have not been systematically compared, and it is not understood how they interact with different morphological typologies. On a language modeling task, we present ex- periments that systematically vary (1) the basic unit of representation, (2) the com- position of these representations, and (3) the morphological typology of the lan- guage modeled. Our results extend previ- ous findings that character representations are effective across typologies, and we find that a previously unstudied combination of character trigram representations com- posed with bi-LSTMs outperforms most others. But we also find room for improve- ment: none of the character-level models match the predictive accuracy of a model with access to true morphological analy- ses, even when learned from an order of magnitude more data. 1 Introduction Continuous representations of words learned by neural networks are central to many NLP tasks (Cho et al., 2014; Chen and Manning, 2014; Dyer et al., 2015). However, directly mapping a fi- nite set of word types to a continuous representa- tion has well-known limitations. First, it makes a closed vocabulary assumption, enabling only generic out-of-vocabulary handling. Second, it cannot exploit systematic functional relationships in learning. For example, cat and cats stand in the same relationship as dog and dogs. While this re- lationship might be discovered for these specific frequent words, it does not help us learn that the same relationship also holds for the much rarer words sloth and sloths. These functional relationships reflect the fact that words are composed from smaller units of meaning, or morphemes. For instance, cats con- sists of two morphemes, cat and -s, with the latter shared by the words dogs and tarsiers. Modeling this effect is crucial for languages with rich mor- phology, where vocabulary sizes are larger, many more words are rare, and many more such func- tional relationships exist. Hence, some models produce word representations as a function of sub- word units obtained from morphological segmen- tation or analysis (Luong et al., 2013; Botha and Blunsom, 2014; Cotterell and Sch¨utze, 2015). A downside of these models is that they depend on morphological segmenters or analyzers. Morphemes typically have similar orthographic representations across words. For example, the morpheme -s is realized as -es in finches. Since this variation is limited, the general relationship between morphology and orthography …

Content truncated to 500 words