Determining Semantic Textual Similarity using Natural Deduction Proofs

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 681–691 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Determining Semantic Textual Similarity using Natural Deduction Proofs Hitomi Yanaka1 hitomiyanaka@g.ecc.u-tokyo.ac.jp Koji Mineshima2 mineshima.koji@ocha.ac.jp Pascual Mart´ınez-G´omez3 pascual.mg@aist.go.jp Daisuke Bekki2 bekki@is.ocha.ac.jp 1The University of Tokyo 2Ochanomizu University 3Artificial Intelligence Research Center, AIST Tokyo, Japan Abstract Determining semantic textual similarity is a core research subject in natural language processing. Since vector-based models for sentence representation often use shallow information, capturing accurate semantics is difficult. By contrast, logical seman- tic representations capture deeper levels of sentence semantics, but their symbolic na- ture does not offer graded notions of tex- tual similarity. We propose a method for determining semantic textual similarity by combining shallow features with features extracted from natural deduction proofs of bidirectional entailment relations between sentence pairs. For the natural deduc- tion proofs, we use ccg2lambda, a higher- order automatic inference system, which converts Combinatory Categorial Gram- mar (CCG) derivation trees into semantic representations and conducts natural de- duction proofs. Experiments show that our system was able to outperform other logic- based systems and that features derived from the proofs are effective for learning textual similarity. 1 Introduction Determining semantic textual similarity (STS) is one of the most critical tasks in informa- tion retrieval and natural language processing. Vector-based sentence representation models have been widely used to compare and rank words, phrases or sentences using various similarity and relatedness scores (Wong and Raghavan, 1984; Mitchell and Lapata, 2010; Le and Mikolov, 2014). Recently, neural network-based sentence representation models (Mueller and Thyagarajan, 2016; Hill et al., 2016) have been proposed for learning textual similarity. However, these vector- based models often use shallow information, such as words and characters, and whether they can account for phenomena such as negation and quantification is not clear. Consider the sentences: Tom did not meet some of the players and Tom did not meet any of the players. If functional words such as some or any are ignored or represented as the same vector, then these sentences are to be represented by identical vectors. However, the first sentence implies that there is a player who Tom did not meet, whereas the second sentence means that Tom did not meet anyone, so the sentences have different meanings. Conversely, logic-based approaches have been successful in representing the meanings of com- plex sentences, having had a positive impact for applications such as recognizing textual entail- ment (Mineshima et al., 2015, 2016; Abzianidze, 2015, 2016). However, purely logic-based ap- proaches only assess entailment or contradic- tion relations between sentences and do not offer graded notions of semantic similarity. In this paper, we propose to leverage logic cues to learn textual similarity. Our hypothesis is that observing proof processes when testing the seman- tic relations is predictive of textual similarity. We show that our approach can be more effective than systems that ignore these logic cues. 2 Related Work Vector-based models of semantic composition have been widely studied with …

Content truncated to 500 words