Do RNN States Encode Abstract Phonological Alternations?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5501–5513 June 6–11, 2021. ©2021 Association for Computational Linguistics 5501 Do RNN States Encode Abstract Phonological Alternations? Miikka SilfverbergZ Francis TyersQ Garrett NicolaiZ Mans HuldenX ZUniversity of British Columbia QIndiana University XUniversity of Colorado first.last@ubc.ca ftyers@iu.edu first.last@colorado.edu Abstract Sequence-to-sequence models have delivered impressive results in word formation tasks such as morphological inflection, often learn- ing to model subtle morphophonological de- tails with limited training data. Despite the performance, the opacity of neural models makes it difficult to determine whether com- plex generalizations are learned, or whether a kind of separate rote memorization of each morphophonological process takes place. To investigate whether complex alternations are simply memorized or whether there is some level of generalization across related sound changes in a sequence-to-sequence model, we perform several experiments on Finnish con- sonant gradation—a complex set of sound changes triggered in some words by certain suffixes. We find that our models often— though not always—encode 17 different con- sonant gradation processes in a handful of di- mensions in the RNN. We also show that by scaling the activations in these dimensions we can control whether consonant gradation oc- curs and the direction of the gradation. 1 Introduction Recent work on computational morphology demon- strates that neural networks can very effectively learn to inflect words, given adequate amounts of training data (Cotterell et al., 2016, 2017). How- ever, in computational morphology and in NLP at large, the interpretability of neural models remains a serious concern (Doshi-Velez and Kim, 2017)—it is unclear how networks trained to inflect words actually accomplish their task. It is also unclear to which extent networks are able to learn linguistic generalizations from their input data instead of sim- ply memorizing training examples and exhibiting a kind of nearest-neighbor behavior. In this paper, we shed light on what kind of linguistic generalizations neural networks are ca- pable of learning from data. We report on an in- 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 Figure 1: Scatter plot of the activation for two encoder hidden state dimensions which activate strongly dur- ing gradation. The letters p, t and k refer to examples which undergo gradation for the respective consonants. Lower-case characters mark direct (weakening) grada- tion and upper-case characters inverse (strengthening) gradation (see Section 3 for further details). Black dots · mark examples which do not undergo gradation. vestigation into how consonant gradation, a par- ticular morphophonological alternation which is common in Finnish and other Uralic languages, is encoded in the hidden states of an LSTM encoder- decoder model trained to perform word inflection. Specifically, we train character-based sequence-to- sequence models for inflection of Finnish nouns into the genitive case, an inflection type which com- monly triggers consonant gradation. Consonant gradation is a morphophonological alternation where voiceless stops p, t and k are lenited in certain positions (see Section 3 for fur- …

Content truncated to 500 words