Do latent tree learning models identify meaningful structure in sentences?

Do latent tree learning models identify meaningful structure in sentences? Adina Williams1 adinawilliams@nyu.edu Andrew Drozdov2,∗ andrew.drozdov@nyu.edu Samuel R. Bowman1,2,3 bowman@nyu.edu 1Dept. of Linguistics New York University 10 Washington Place New York, NY 10003 2Dept. of Computer Science New York University 60 Fifth Avenue New York, NY 10011 3Center for Data Science New York University 60 Fifth Avenue New York, NY 10011 Abstract Recent work on the problem of latent tree learning has made it possible to train neu- ral networks that learn to both parse a sen- tence and use the resulting parse to interpret the sentence, all without exposure to ground- truth parse trees at training time. Surprisingly, these models often perform better at sentence understanding tasks than models that use parse trees from conventional parsers. This paper aims to investigate what these latent tree learn- ing models learn. We replicate two such mod- els in a shared codebase and find that (i) only one of these models outperforms conventional tree-structured models on sentence classifica- tion, (ii) its parsing strategies are not espe- cially consistent across random restarts, (iii) the parses it produces tend to be shallower than standard Penn Treebank (PTB) parses, and (iv) they do not resemble those of PTB or any other semantic or syntactic formalism that the authors are aware of. 1 Introduction Tree-structured recursive neural networks (TreeRNNs; Socher et al., 2011)—which build a vector representation for a sentence by incremen- tally computing representations for each node in its parse tree—have been proven to be effective at sentence understanding tasks like sentiment analysis (Socher et al., 2013), textual entailment (Bowman et al., 2016), and translation (Eriguchi et al., 2016). Some variants of these models (Socher et al., 2011; Bowman et al., 2016) can also be trained to produce ∗Now at eBay, Inc. I saw the man with the telescope I saw the man with the telescope (a) Two parse trees correspond to two distinct interpretations for the sentence in example (1). He swung at the brute with his sword . He swung at the brute with his sword . (b) Parses generated by at ST-Gumbel model (left) and the Stan- ford Parser (right). Figure 1: Examples of unlabeled binary parse trees. parse trees that they then consume. Recent work on latent tree learning (Yogatama et al., 2017; Maillard et al., 2017; Choi et al., 2018) has led to the development of new training methods for TreeRNNs that allow them to learn to parse without ever being given an example of a correct parse tree, thereby replacing direct syntactic supervision with indirect supervision from a downstream task like sentence classification. These models are designed to learn grammars—strategies for assigning trees to sentences—that are suited to help solve the sentence understanding task at hand, rather than ones that approximate expert-designed grammars like that of the Penn Treebank (PTB; Marcus et al., 1999). Latent tree learning models have shown striking success at sentence understanding, reliably perform- ing better on sentiment analysis and textual entail- ment than do comparable …

Content truncated to 500 words