Empirical Investigation of Neural Symbolic Reasoning Strategies

Findings of the Association for Computational Linguistics: EACL 2023, pages 1154–1162 May 2-6, 2023 ©2023 Association for Computational Linguistics Empirical Investigation of Neural Symbolic Reasoning Strategies Yoichi Aoki,1 Keito Kudo,1 Tatsuki Kuribayashi,1,2 Ana Brassard,3,1 Masashi Yoshikawa,1 Keisuke Sakaguchi,1,3 Kentaro Inui1,3 1Tohoku University, 2Langsmith, Inc., 3RIKEN {youichi.aoki.p2,keito.kudo.q4}@dc.tohoku.ac.jp, kuribayashi@tohoku.ac.jp, ana.brassard@riken.jp, {yoshikawa,keisuke.sakaguchi,kentaro.inui}@tohoku.ac.jp Abstract Neural reasoning accuracy improves when gen- erating intermediate reasoning steps. However, the source of this improvement is yet unclear. Here, we investigate and factorize the benefit of generating intermediate steps for symbolic reasoning. Specifically, we decompose the reasoning strategy w.r.t. step granularity and chaining strategy. With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the per- formance, with the gap becoming even larger as the extrapolation length becomes longer. Surprisingly, we also found that certain configu- rations lead to nearly perfect performance, even in the case of length extrapolation. Our results indicate the importance of exploring effective strategies for neural reasoning models. 1 1 Introduction Artificial intelligence researchers have been at- tempting neural-symbolic integration for a long time (d’Avila Garcez and Lamb, 2020; Hamilton et al., 2022). Neural models tend to perform better when generating intermediate reasoning steps in addition to the answer. This phenomenon was seen across various reasoning tasks, such as math word problems (Wei et al., 2022; Cobbe et al., 2021; Ko- jima et al., 2022; Recchia, 2021; Lewkowycz et al., 2022), commonsense reasoning (Wei et al., 2022; Wang et al., 2022), and symbolic reasoning (Wei et al., 2022; Kojima et al., 2022). However, it is yet unclear which factors in the intermediate step generation bring the benefit. Previous studies often used different strategies for step generation in an ad- hoc manner. To investigate this, we break down the neural reasoning process into two strategies: output strategy and chaining strategy (Figure 1). The output strategy (§2.1) determines the granularity of 1Code available at: https://github.com/ao1neko/ reasoning-strategy Figure 1: In a controlled setting, we found that output and chaining strategy choice significantly impact performance when conducting multi-step reasoning. intermediate reasoning step generation (all at once vs. step-by-step vs. token-by-token). Some studies trained the models to generate reasoning steps and a conclusion derived from them at once (Nye et al., 2021; Lewkowycz et al., 2022; Wei et al., 2022; Kojima et al., 2022; Wang et al., 2022; Recchia, 2021), some generated a single reasoning step given the input and iterated this process until achieving a conclusion (Sanyal et al., 2022; Picco et al., 2021; Tafjord et al., 2021), and others iteratively generated sub-goals as well as reasoning steps (Liang et al., 2021; Shwartz et al., 2020). In turn, the chaining strategy (§2.2) defines the reasoning path direction (shortest path vs. exhaustive path vs. backward path). For ex- ample, some studies used a backward chaining process (Picco et al., 2021; Rocktäschel and Riedel, 2017; Cingillioglu and Russo, 2019), while others adopted exhaustive searches (Tafjord et al., 2021; Liang et al., 2021; Yang et al., 2022). To compare the …

Content truncated to 500 words