When Are Tree Structures Necessary for Deep Learning of Representations?

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2304–2314, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. When Are Tree Structures Necessary for Deep Learning of Representations? Jiwei Li1, Minh-Thang Luong1, Dan Jurafsky1 and Eduard Hovy2 1Computer Science Department, Stanford University, Stanford, CA 94305 2Language Technology Institute, Carnegie Mellon University, Pittsburgh, PA 15213 jiweil,lmthang,jurafsky@stanford.edu ehovy@andrew.cmu.edu Abstract Recursive neural models, which use syn- tactic parse trees to recursively generate representations bottom-up, are a popular architecture. However there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper, we benchmark recursive neural models against sequential recurrent neural models, enforcing apples- to-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment clas- sification at the sentence level and phrase level; (2) matching questions to answer- phrases; (3) discourse parsing; (4) seman- tic relation extraction. Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like seman- tic relation extraction) that require long- distance connection modeling, particularly on very long sequences. We then intro- duce a method for allowing recurrent mod- els to achieve similar performance: break- ing long sentences into clause-like units at punctuation and processing them sepa- rately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models. 1 Introduction Deep learning based methods learn low- dimensional, real-valued vectors for word tokens, mostly from large-scale data corpus (e.g., (Mikolov et al., 2013; Le and Mikolov, 2014; Collobert et al., 2011)), successfully capturing syntactic and semantic aspects of text. For tasks where the inputs are larger text units (e.g., phrases, sentences or documents), a compo- sitional model is first needed to aggregate tokens into a vector with fixed dimensionality that can be used as a feature for other NLP tasks. Models for achieving this usually fall into two categories: re- current models and recursive models: Recurrent models (also referred to as sequence models) deal successfully with time-series data (Pearlmutter, 1989; Dorffner, 1996) like speech (Robinson et al., 1996; Lippmann, 1989; Graves et al., 2013) or handwriting recognition (Graves and Schmidhuber, 2009; Graves, 2012). They were ap- plied early on to NLP (Elman, 1990), by modeling a sentence as tokens processed sequentially and at each step combining the current token with pre- viously built embeddings. Recurrent models can be extended to bidirectional ones from both left- to-right and right-to-left. These models generally consider no linguistic structure aside from word order. Recursive neural models (also referred to as tree models), by contrast, are structured by syntactic parse trees. Instead of considering tokens sequen- tially, recursive models combine neighbors based on the recursive structure of parse trees, starting from the leaves and proceeding recursively in a bottom-up fashion until the root of the parse tree is reached. For example, for the phrase the food is delicious, following the operation sequence ( (the food) (is …

Content truncated to 500 words