E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning

Findings of the Association for Computational Linguistics: ACL 2022, pages 3941 - 3955 May 22-27, 2022 c⃝2022 Association for Computational Linguistics E-KAR : A Benchmark for Rationalizing Natural Language Analogical Reasoning Jiangjie Chen♠♦∗, Rui Xu♠, Ziquan Fu♥, Wei Shi♣, Zhongqiao Li♠, Xinbo Zhang♦, Changzhi Sun♦†, Lei Li¶, Yanghua Xiao♠§†, Hao Zhou♦ ♠Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University ♦ByteDance AI Lab ♥Brain Technologies, Inc. ♣South China University of Technology ¶University of California Santa Barbara §Fudan-Aishu Cognitive Intelligence Joint Research Center {jjchen19,shawyh}@fudan.edu.cn, sunchangzhi@bytedance.com Abstract The ability to recognize analogies is funda- mental to human cognition. Existing bench- marks to test word analogy do not reveal the underneath process of analogical reasoning of neural models. Holding the belief that mod- els capable of reasoning should be right for the right reasons, we propose a first-of-its- kind Explainable Knowledge-intensive Ana- logical Reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (in Chinese) and 1,251 (in English) problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. More im- portantly, we design a free-text explanation scheme to explain whether an analogy should be drawn, and manually annotate them for each and every question and candidate answer. Empirical results suggest that this benchmark is very challenging for some state-of-the-art models for both explanation generation and analogical question answering tasks, which in- vites further research in this area. Project page of E-KAR can be found at https:// ekar-leaderboard.github.io. 1 Introduction Analogy holds a vital place in human cognition, driving the discovery of new insights and the jus- tification of everyday reasoning (Johnson-Laird, 2006; Gentner and Smith, 2012; Bartha, 2013; Ben- gio et al., 2021). Due to their unique value in many fields such as creativity (Goel, 1997) and education (Thagard, 1992), analogy and analogical reasoning have become a focus in AI research. The grand question is, are artificial neural networks also capa- ble of recognizing analogies? Relatively little attention has been paid in NLP to answer this question. The problem of recogniz- ing analogies is mainly benchmarked in the form ∗Work is done during internship at ByteDance AI Lab. †Corresponding authors. Both “teapot” and “teacup” are containers for holding “tea” . After the “tea” is brewed in the “teapot” , it is transported into the “teacup” . 2 3 1 1 2 3 tea :teapot :teacup 1 2 3 Q) textbooks :bookstore :printing factory 1 2 3 D) bookstore2 printing factory3 transport textbooks1 bookstore2 printing factory3 is_aorganization is_a After “textbooks” are printed in the “printing factory”, they are sold in a “bookstore”. But the terms order is inconsistent with the query. passengers :bus :taxi 1 2 3 A) bus2 taxi3 transportation for passengers1 is_a is_a bus2 taxi3 transport passengers1 “Passengers” do not need to be transported into “taxi” after taking a “bus”. “Taxi” and “bus” are different ways of transportation. magazine :bookshelf :reading room 1 2 3 B) bookshelf2 reading room3 ? is_a is_a The “bookshelf” is in the “reading room”. talents :school :enterprise 1 2 3 C) …

Content truncated to 500 words