‘Indicatements’ that character language models learn English morpho-syntactic units and regularities

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 145–153 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 145 Indicatements that character language models learn English morpho-syntactic units and regularities Yova Kementchedjhieva University of Copenhagen yova@di.ku.dk Adam Lopez University of Edinburgh alopez@inf.ed.ac.uk Abstract Character language models have access to sur- face morphological patterns, but it is not clear whether or how they learn abstract morpho- logical regularities. We instrument a charac- ter language model with several probes, find- ing that it can develop a specific unit to iden- tify word boundaries and, by extension, mor- pheme boundaries, which allows it to capture linguistic properties and regularities of these units. Our language model proves surpris- ingly good at identifying the selectional re- strictions of English derivational morphemes, a task that requires both morphological and syntactic awareness. Thus we conclude that, when morphemes overlap extensively with the words of a language, a character language model can perform morphological abstraction. 1 Introduction Character-level language models (Sutskever et al., 2011) are appealing because they enable open- vocabulary generation of language, and condi- tional character language models have now been convincingly used in speech recognition (Chan et al., 2016) and machine translation (Weiss et al., 2017; Lee et al., 2016; Chung et al., 2016). They succeed due to parameter-sharing between fre- quent, rare, and even unobserved training words, prompting claims that they learn morphosyntac- tic properties of words. For example, Chung et al. (2016) claim that character language mod- els yield “better modelling [of] rare morpholog- ical variants” while Kim et al. (2016) claim that “Character-level models obviate the need for mor- phological tagging or manual feature engineer- ing.” But these claims of morphological awareness are backed more by intuition than direct empirical evidence. What do these models really learn about morphology? And, to the extent that they learn about morphology, how do they learn it? Our goal is to shed light on these questions, and to that end, we study the behavior of a character- level language model (hereafter LM) applied to English. We observe that, when generating text, the LM applies certain morphological processes of English productively, i.e. in novel contexts (§3). This rather surprising finding suggests that the model can identify the morphemes relevant to these processes. An analysis of the LM’s hidden units presents a possible explanation: there ap- pears to be one particular unit that fires at mor- pheme and word boundaries (§4). Further experi- ments reveal that the LM learns morpheme bound- aries through extrapolation from word boundaries (§5). In addition to morphology, the LM appears to encode syntactic information about words, i.e. their part of speech (§6). With access to both morphology and syntax, the model should also be able to learn linguistic phenomena at the intersec- tion of the two domains, which we indeed find to be the case: the LM captures the (syntactic) se- lectional restrictions of English derivational mor- phemes, albeit with some incorrect generalizations (§7). The …

Content truncated to 500 words