Discourse Parsing with Attention-based Hierarchical Neural Networks

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 362–371, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Discourse Parsing with Attention-based Hierarchical Neural Networks Qi Li Tianshi Li Baobao Chang Key Laboratory of Computational Linguistics, Ministry of Education School of Electronics Engineering and Computer Science, Peking University No.5 Yiheyuan Road, Haidian District, Beijing, 100871, China Collaborative Innovation Center for Language Ability, Xuzhou, 221009, China qi.li@pku.edu.cn lts 417@hotmail.com chbb@pku.edu.cn Abstract RST-style document-level discourse parsing remains a difficult task and efficient deep learning models on this task have rarely been presented. In this paper, we propose an attention-based hierarchical neural network model for discourse parsing. We also incor- porate tensor-based transformation function to model complicated feature interactions. Ex- perimental results show that our approach ob- tains comparable performance to the contem- porary state-of-the-art systems with little man- ual feature engineering. 1 Introduction A document is formed by a series of coherent text units. Document-level discourse parsing is a task to identify the relations between the text units and to determine the structure of the whole document the text units form. Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) is one of the most in- fluential discourse theories. According to RST, the discourse structure of a document can be represented by a Discourse Tree (DT). Each leaf of a DT denotes a text unit referred to as an Elementary Discourse Unit (EDU) and an inner node of a DT represents a text span which is constituted by several adjacent EDUs. DTs can be utilized by many NLP tasks in- cluding automatic document summarization (Louis et al., 2010; Marcu, 2000), question-answering (Ver- berne et al., 2007) and sentiment analysis (Somasun- daran, 2010) etc. Much work has been devoted to the task of RST- style discourse parsing and most state-of-the-art ap- proaches heavily rely on manual feature engineer- ing (Joty et al., 2013; Feng and Hirst, 2014; Ji and Eisenstein, 2014). While neural network mod- els have been increasingly focused on for their abil- ity to automatically extract efficient features which reduces the burden of feature engineering, there is little neural network based work for RST-style dis- course parsing except the work of Li et al. (2014a). Li et al. (2014a) propose a recursive neural network model to compute the representation for each text span based on the representations of its subtrees. However, vanilla recursive neural networks suffer from gradient vanishing for long sequences and the normal transformation function they use is weak at modeling complicated interactions which has been stated by Socher et al. (2013). As many docu- ments contain more than a hundred EDUs which form quite a long sequence, those weaknesses may lead to inferior results on this task. In this paper, we propose to use a hierarchical bidirectional Long Short-Term Memory (bi-LSTM) network to learn representations of text spans. Com- paring with vanilla recursive/recurrent neural net- works, LSTM-based networks can store information for a long period of time and don’t suffer from gra- dient …

Content truncated to 500 words