Natural Language Learning by Recurrent Neural Networks: A Comparison wRh probabilistic approaches

l / / / / I / / / / / / / / Natural Language Learning by Recurrent Neural Networks: A Comparison with probabilistic approaches Michael Towsey*, Joachim Diederich*, lngo Schellhammer*#, Stephan Chalup*, Claudia Brugman** * Neurocomputing Research Centre, Queensland University of Technology, QLD, 4001, Australia # Dept of Information Systems, University of Muenster, D-48149 Muenster, Germany **School of Languages, University ofOtago, Dunedin, New Zealand j oachim@fit, qut. edu. au Abstract We present preliminary results of experiments with two types of recurrent neural networks for a natural language learning task. The neural networks, Elman networks and Recurrent Cascade Correlation (RCC), were trained on the text of a first-year primary school reader. The networks performed a one-step-look-ahead task, i.e. they had to predict the lexical category of the next following word. Elman networks with 9 hidden units gave the best training results (72% correct) but scored only 63% when tested for generalisation using a "leave-one-sentence-out" cross-validation technique. An RCC network could learn 99.6% of the training set by adding up to 42 hidden units but achieved best generalisation (63%) with only four hidden units. Results are presented showing network learning in relation to bi-, t'i-, 4- and 5-gram performance. Greatest prediction uncertainty (measured as the entropy of the output units) occurred, not at the sentence boundaries but when the first verb was the input. 1. Introduction Elman (1990) emphasised that natural language input unfolds in time and therefore, recurrent networks which can accept a sequence of input patterns are the preferred choice for many connectionist natural language processing tasks. In recurrent networks, knowledge is represented in activation patterns over hidden units and revealed (i.e. made explicit) by hierarchical cluster analysis or other statistical methods. Furthermore, recent evidence from cognitive neuroscience (Singer, 1995) points to the importance of recurrent connections for the formation of coherent cell assemblies. Recent work on recurrent neural networks has focussed on formal languages (Wiles & Elman, 1995). In this paper, we present preliminary results of experiments with recun'ent neural networks for a natural language learning task. Our strategy is to start with simple children's texts and to step-wise increase the complexity of these texts to explore the learning characteristics of recurrent neural networks. In the first experiments reported here, we are starting with a first- year primary school reader from which sentences with embedded structures have been diminatecL In future experiments, we will use unmodified first-year texts and will continue with second-year textbooks and so on. 1.1 Elman and RCC networks Simple recurrent networks (SRN's) of the Elman type are similar to three-layer perceptrons but with recurrent connections from the hidden layer to a context layer (also called state layer) which becomes part of the input. The activation patterns of the hidden units at time step t are copied onto the context units and presented with the input at the next time step. The hidden units have the task of mapping both an external input, and also the previous internal state in order to produce some …

Content truncated to 500 words