Analyzing Linguistic Knowledge in Sequential Model of Sentence

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 826–835, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Analyzing Linguistic Knowledge in Sequential Model of Sentence Peng Qian Xipeng Qiu∗ Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {pqian11, xpqiu, xjhuang}@fudan.edu.cn Abstract Sentence modelling is a fundamental top- ic in computational linguistics. Recently, deep learning-based sequential models of sen- tence, such as recurrent neural network, have proved to be effective in dealing with the non-sequential properties of human language. However, little is known about how a recurrent neural network captures linguistic knowledge. Here we propose to correlate the neuron activation pattern of a LSTM language model with rich language features at sequential, lexical and compositional level. Qualitative visualization as well as quantitative analy- sis under multilingual perspective reveals the effectiveness of gate neurons and indicates that LSTM learns to allow different neurons selectively respond to linguistic knowledge at different levels. Cross-language evidence shows that the model captures different as- pects of linguistic properties for different languages due to the variance of syntactic complexity. Additionally, we analyze the influence of modelling strategy on linguistic knowledge encoded implicitly in different sequential models. 1 Introduction Sentence modelling is a central and fundamental topic in the study of language generation and comprehension. With the application of popular deep learning methods, researchers have found that recurrent neural network can successfully model the non-sequential linguistic properties with sequential ∗Corresponding author. data input (Vinyals et al., 2015; Zhou and Xu, 2015; Rockt¨aschel et al., 2015). However, due to the complexity of the neural networks and the lack of effective analytic methodology, little is known about how a sequential model of sentence, such as recurrent neural network, captures linguistic knowledge. This makes it hard to understand the underlying mechanism as well as the model’s strength and weakness. Previous work (Li et al., 2016) has attempted to visualize neural models in NLP, but only focus on analyzing the hidden layer and sentiment representation rather than grammar knowledge. Currently, there have been a few attempts (Yo- gatama et al., 2014; K¨ohn, 2015; Faruqui and Dyer, 2015) at understanding what is embedded in the word vectors or building linguistically interpretable embeddings. Few works focus on investigating the linguistic knowledge encoded in a sequential neural network model of a sentence, not to mention the comparison of model behaviours from a cross- language perspective. Our work, therefore, aims to shedding new insights into the following topics: a) How well does a sequential neural model (e.g. language model) encodes linguistic knowledge of different levels? b) How does modelling strategy (e.g. the optimiza- tion objective) influence the neuron’s ability of capturing linguistic knowledge? c) Does the sequential model behave similarly to- wards typologically diverse languages? To tackle the questions above, we propose to visualize and analyze the neuron activation pattern 826 est transformé Model ··· · · · Activati Input Sequence en ,1868 le journal …

Content truncated to 500 words