Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering

Proceedings of the 29th International Conference on Computational Linguistics, pages 1765–1775 October 12–17, 2022. 1765 Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering Zhengbao Jiang†, Jun Araki‡, Haibo Ding‡∗, Graham Neubig† †Languages Technologies Institute, Carnegie Mellon University ‡Bosch Research {zhengbaj,gneubig}@cs.cmu.edu {jun.araki,haibo.ding}@us.bosch.com Abstract Generative question answering (QA) models generate answers to questions either solely based on the parameters of the model (the closed-book setting) or additionally retriev- ing relevant evidence (the open-book setting). Generative QA models can answer some rela- tively complex questions, but the mechanism through which they do so is still poorly under- stood. We perform several studies aimed at better understanding the multi-hop reasoning capabilities of generative QA models. First, we decompose multi-hop questions into mul- tiple corresponding single-hop questions, and find marked inconsistency in QA models’ an- swers on these pairs of ostensibly identical question chains. Second, we find that mod- els lack zero-shot multi-hop reasoning abil- ity: when trained only on single-hop questions, models generalize poorly to multi-hop ques- tions. Finally, we demonstrate that it is pos- sible to improve models’ zero-shot multi-hop reasoning capacity through two methods that approximate real multi-hop natural language (NL) questions by training on either concate- nation of single-hop questions or logical forms (SPARQL). In sum, these results demonstrate that multi-hop reasoning does not emerge nat- urally in generative QA models, but can be en- couraged by advances in training or modeling techniques.1 1 Introduction Empowered by large-scale pre-trained language models (LMs) (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020a; Raffel et al., 2020), recent years have seen much progress on generative ques- tion answering (QA), where LMs generate answers given questions in an end-to-end fashion. While most works only demonstrate the performance of such generative QA models on simple questions (Joshi et al., 2017; Kwiatkowski et al., 2019), there ∗Haibo Ding is now at Amazon. 1Code is available at https://github.com/jzbjyb/multihop. Closed-book QA model !! Return the artist who recorded Party Ain't Over. !" Where in Georgia does Usher live? ! Which part of Georgia does the artist that recorded Party Ain't Over live? Open-book QA model "! "! "" "" Questions Contexts #$! Rihanna #$" Atlanta #$ Atlanta Chris Brown ✗ (Usher) Atlanta ✔ Atlanta ✔ Predictions: Figure 1: Probing generative closed- and open-book QA models with both multi-hop (q) and their compo- nent single-hop questions (q1, q2). has been some indication that these models can also answer complex questions that theoretically require multi-hop reasoning (Xiong et al., 2020), sometimes to an impressive degree. For exam- ple, Brown et al. (2020) demonstrate strong perfor- mance of LMs on multi-hop reasoning tasks such as DROP (Dua et al., 2019) which requires discrete reasoning and numeracy. On the other hand, many argue that LM-based QA models are not actually performing any reasoning, and rather performing (sophisticated) pattern matching and data memo- rization (Marcus and Davis, 2020). Simultaneously, in the context of extractive QA models that select answers from the provided context, several works have demonstrated that …

Content truncated to 500 words