Cooperative Learning of Disjoint Syntax and Semantics

Proceedings of NAACL-HLT 2019, pages 1118–1128 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1118 Cooperative Learning of Disjoint Syntax and Semantics Serhii Havrylov ∗ ILCC, University of Edinburgh / Edinburgh, UK s.havrylov@ed.ac.uk Germ´an Kruszewski & Armand Joulin Facebook AI Research {germank,ajoulin}@fb.com Abstract There has been considerable attention devoted to models that learn to jointly infer an ex- pression’s syntactic structure and its seman- tics. Yet, Nangia and Bowman (2018) has re- cently shown that the current best systems fail to learn the correct parsing strategy on math- ematical expressions generated from a sim- ple context-free grammar. In this work, we present a recursive model inspired by Choi et al. (2018) that reaches near perfect accu- racy on this task. Our model is composed of two separated modules for syntax and se- mantics. They are cooperatively trained with standard continuous and discrete optimisation schemes. Our model does not require any lin- guistic structure for supervision, and its re- cursive nature allows for out-of-domain gen- eralisation. Additionally, our approach per- forms competitively on several natural lan- guage tasks, such as Natural Language Infer- ence and Sentiment Analysis. 1 Introduction Standard linguistic theories propose that natural language is structured as nested constituents or- ganised in the form of a tree (Partee et al., 1990). However, most popular models, such as the Long Sort-Term Memory network (LSTM) (Hochreiter and Schmidhuber, 1997), process text without im- posing a grammatical structure. To bridge this gap between theory and practice models that process linguistic expressions in a tree-structured manner have been considered in recent work (Socher et al., 2013; Tai et al., 2015; Zhu et al., 2015; Bowman et al., 2016). These tree-based models explicitly require access to the syntactic structure for the text, which is not entirely satisfactory. Indeed, parse tree level supervision requires a significant amount of annotations from expert lin- ∗Work done while the author was an intern at Facebook AI Research. guists. These trees have been annotated with dif- ferent goals in mind than the tasks we are using them for. Such discrepancy may result in a de- terioration of the performance of models relying on them. Recently, several attempts were made to learn these models without explicit supervi- sion for the parser (Yogatama et al., 2016; Mail- lard et al., 2017; Choi et al., 2018). However, Williams et al. (2018a) has recently shown that the structures learned by these models cannot be as- cribed to discovering meaningful syntactic struc- ture. These models even fail to learn the simple context-free grammar of nested mathematical op- erations (Nangia and Bowman, 2018). In this work, we present an extension of Choi et al. (2018), that successfully learns these simple grammars while preserving competitive perfor- mance on several standard linguistic tasks. Con- trary to previous work, our model makes a clear distinction between the parser and the composi- tional function. These two modules are trained with different algorithms, cooperating to build a semantic representation that optimises the objec- tive function. …

Content truncated to 500 words