Language Modeling with a General Second-Order RNN

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4749–4753 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4749 Language Modeling with a General Second-Order RNN Diego Maupom´e and Marie-Jean Meurs Universit´e du Qu´ebec `a Montr´eal - UQAM Montreal, QC, Canada maupome.diego@courrier.uqam.ca, meurs.marie-jean@uqam.ca Abstract Different Recurrent Neural Network (RNN) architectures update their state in different manners as the input sequence is processed. RNNs including a multiplicative interaction between their current state and the current input, second-order ones, show promising performance in language modeling. In this paper, we introduce a second-order RNNs that generalizes existing ones. Evaluating on the Penn Treebank dataset, we analyze how its different components affect its performance in character-lever recurrent language modeling. We perform our experiments controlling the parameter counts of models. We find that removing the first-order terms does not hinder performance. We perform further experiments comparing the effects of the relative size of the state space and the multiplicative interaction space on performance. Our expectation was that a larger states would benefit language models built on longer documents, and larger multiplicative interaction states would benefit ones built on larger input spaces. However, our results suggest that this is not the case and the optimal relative size is the same for both document tokenizations used. Keywords: Recurrent Neural Networks, Language Modeling 1. Introduction One of the principal challenges in computational linguis- tics is learning probability distributions over sequences of words or characters. Of course, the number of possible phrases grows exponentially with respect to the length. It is therefore required of natural language models to take efficient approaches to representing natural language obser- vations. Recurrent Neural Networks (RNNs) are an elegant solution to this problem. Indeed, by consuming sequences of tokens (e.g. words, characters) in order, RNNs dynamically con- struct an internal representation of the input sentence. This representation can be used for downstream predictions. As for inference, the backpropagation algorithm can intu- itively be extended to run through the time steps of a se- quence in reverse order. This is known as Backpropagation Through Time (BPTT). It allows for efficient computation of the gradients needed for gradient descent. It remains, however, that the shape of an RNN will impact its ability to learn language models. In this work, we take interest in second-order RNN architectures. We begin by introducing the simple first-order RNN as well as several examples of second-order RNNs in Section 2. We then introduce in Section 3. an architecture generalizing these second-order architectures. Section 4. presents our experiments in character-level language modeling, and their results are discussed in Section 5. Finally, Section 6. concludes this paper. 2. First and Second-Order RNNs RNNs process sequential data by reading one token at a time and updating a real-valued vector called their hidden state. Let a document be a sequence x1, . . . , xT indexed by t = 1, . . . , T. The hidden state of the RNN is first null, h0 …

Content truncated to 500 words