Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5508–5521 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5508 Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming Caio Corro Ivan Titov ILCC, School of Informatics, University of Edinburgh ILLC, University of Amsterdam c.f.corro@uva.nl ititov@inf.ed.ac.uk Abstract We treat projective dependency trees as latent variables in our probabilistic model and induce them in such a way as to be beneficial for a downstream task, without relying on any direct tree supervision. Our approach relies on Gum- bel perturbations and differentiable dynamic programming. Unlike previous approaches to latent tree learning, we stochastically sample global structures and our parser is fully differ- entiable. We illustrate its effectiveness on sen- timent analysis and natural language inference tasks. We also study its properties on a syn- thetic structure induction task. Ablation stud- ies emphasize the importance of both stochas- ticity and constraining latent structures to be projective trees. 1 Introduction Discrete structures are ubiquitous in the study of natural languages, for example in morphology, syntax and discourse analysis. In natural language processing, they are often used to inject linguistic prior knowledge into statistical models. For exam- ples, syntactic structures have been shown benefi- cial in question answering (Cui et al., 2005), senti- ment analysis (Socher et al., 2013), machine trans- lation (Bastings et al., 2017) and relation extrac- tion (Liu et al., 2015), among others. However, linguistic tools producing these structured repre- sentations (e.g., syntactic parsers) are not avail- able for many languages and not robust when ap- plied outside of the domain they were trained on (Petrov et al., 2010; Foster et al., 2011). More- over, linguistic structures do not always seem suitable in downstream applications, with sim- pler alternatives sometimes yielding better perfor- mance (Wang et al., 2018). Indeed, a parallel line of work focused on induc- ing task-specific structured representations of lan- guage (Naradowsky et al., 2012; Yogatama et al., 2017; Kim et al., 2017; Liu and Lapata, 2018; Nic- ulae et al., 2018). In these approaches, no syn- tactic or semantic annotation is needed for train- ing: representation is induced from scratch in an end-to-end fashion, in such a way as to benefit a given downstream task. In other words, these ap- proaches provide an inductive bias specifying that (hierarchical) structures are appropriate for repre- senting a natural language, but do not make any further assumptions regarding what the structures represent. Structures induced in this way, though useful for the task, tend not to resemble any ac- cepted syntactic or semantic formalisms (Williams et al., 2018a). Our approach falls under this cate- gory. In our method, projective dependency trees (see Figure 3 for examples) are treated as latent vari- ables within a probabilistic model. We rely on differentiable dynamic programming (Mensch and Blondel, 2018) which allows for efficient sampling of dependency trees (Corro and Titov, 2019). Intu- itively, sampling a tree involves stochastically per- turbing dependency weights and then running a …

Content truncated to 500 words