Exploring End-to-End Differentiable Natural Logic Modeling

Proceedings of the 28th International Conference on Computational Linguistics, pages 1172–1185 Barcelona, Spain (Online), December 8-13, 2020 1172 Exploring End-to-End Differentiable Natural Logic Modeling Yufei Feng†∗, Zi’ou Zheng†∗, Quan Liu‡, Michael Greenspan†, Xiaodan Zhu† †Electrical and Computer Engineering & Ingenuity Labs Research Institute, Queen’s University ‡State Key Laboratory of Cognitive Intelligence, iFLYTEK Research {feng.yufei,ziou.zheng,michael.greenspan,xiaodan.zhu}@queensu.ca quanliu@iflytek.com Abstract We explore end-to-end trained differentiable models that integrate natural logic with neural net- works, aiming to keep the backbone of natural language reasoning based on the natural logic formalism while introducing subsymbolic vector representations and neural components. The proposed model adapts module networks to model natural logic operations, which is enhanced with a memory component to model contextual information. Experiments show that the proposed framework can effectively model monotonicity-based reasoning, compared to the baseline neural network models without built-in inductive bias for monotonicity-based reasoning. Our proposed model shows to be robust when transferred from upward to downward inference. We perform fur- ther analyses on the performance of the proposed model on aggregation, showing the effectiveness of the proposed subcomponents on helping achieve better intermediate aggregation performance. 1 Introduction A recent research trend has attempted to further advance the long-standing problem of bringing together the complementary strengths of neural networks and symbolic models, e.g., the research performed in (Garcez et al., 2015; Yang et al., 2017; Rockt¨aschel and Riedel, 2017; Evans and Grefenstette, 2018; Weber et al., 2019; De Raedt et al., 2019; Mao et al., 2019), among others. It is known that neural models can approximate complex functions and are robust to noise and ambiguity, while symbolic models often render superior explainability and interpretability but are brittle and prone to fail in the presence of noise and uncertainty. The majority of research efforts are based on some abstract logical forms such as the first-order logic (FOL) or its fragments. For natural language, obtaining such a representation is known to face many thorny challenges. Natural logic instead aims to sidestep some of the challenges by performing inferences over surface forms of text based on monotonicity or projectivity (Van Benthem, 1986; Valencia, 1991; MacCartney and Manning, 2009; Icard and Moss, 2014), and has been applied to tasks such as natural language inference (MacCartney and Manning, 2009; Angeli and Manning, 2014) and question answering (Angeli et al., 2016). In this work we explore differentiable natural logic models that integrate natural logic with neural networks, with the aim to keep the backbone of inference based on the natural logic formalism, while introducing subsymbolic vector representations and neural components into the framework. Combining the advantages of neural networks with natural logic needs to take several basic problems into consideration. Two problems flow directly from this objective: 1) How (and where) to leverage the strength of neural networks in the natural logic formalism, and; 2) How to alleviate the issue of a lack of intermediate supervision for training sub-components, which may lead to the spurious problem (Guu et al., 2017; Min et al., 2019) in the end-to-end training. We explore …

Content truncated to 500 words