Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text

Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text Michael Hahn∗ Stanford University mhahn2@stanford.edu Marco Baroni Facebook AI Research UPF Linguistics Department Catalan Institution for Research and Advanced Studies mbaroni@gmail.com Abstract Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine lin- guistic knowledge. Nearly all current ana- lytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our ‘‘near tabula rasa’’ RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage. 1 Introduction Recurrent neural networks (RNNs; Elman, 1990), in particular in their long short term memory variant (LSTMs; Hochreiter and Schmidhuber, 1997), are widely used in natural language pro- cessing. RNNs, often pre-trained on the simple ∗Work partially done while interning at Facebook AI Research. language modeling objective of predicting the next symbol in natural text, are a crucial com- ponent of state-of-the-art architectures for machine translation, natural language inference, and text categorization (Goldberg, 2017). RNNs are very general devices for sequence processing, hardly assuming any prior linguistic knowledge. Moreover, the simple prediction task they are trained on in language modeling is well- attuned to the core role that prediction plays in cognition (e.g., Bar, 2007; Clark, 2016). RNNs have thus long attracted researchers interested in language acquisition and processing. Their recent success in large-scale tasks has rekindled this interest (e.g., Frank et al., 2013; Lau et al., 2017; Kirov and Cotterell, 2018; Linzen et al., 2018; McCoy et al., 2018; Pater, 2018). The standard pre-processing pipeline of modern RNNs assumes that the input has been tokenized into word units that are pre-stored in the RNN vocabulary (Goldberg, 2017). This is a reasonable practical approach, but it makes simulations less interesting from a linguistic point of view. First, discovering words (or other primitive constituents of linguistic structure) is one of the major chal- lenges a learner faces, and by pre-encoding them in the RNN we are facilitating its task in an unnatural way (not even the staunchest nativists would take specific word dictionaries to be part of our genetic code). Second, assuming a unique tokenization into a finite number of discrete word units is in any case problematic. The very notion of what counts as a word in languages with a rich morphol- ogy is far from clear (e.g., Dixon and Aikhenvald, 2002; Bickel and …

Content truncated to 500 words