Autoregressive Reasoning over Chains of Facts with Transformers

Proceedings of the 28th International Conference on Computational Linguistics, pages 6916–6930 Barcelona, Spain (Online), December 8-13, 2020 6916 Autoregressive Reasoning over Chains of Facts with Transformers Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens LIIR lab, KU Leuven, Belgium {first}.{last}@kuleuven.be, sien.moens@kuleuven.be Abstract This paper proposes an iterative inference algorithm for multi-hop explanation regeneration, that retrieves relevant factual evidence in the form of text snippets, given a natural language ques- tion and its answer. Combining multiple sources of evidence or facts for multi-hop reasoning becomes increasingly hard when the number of sources needed to make an inference grows. Our algorithm copes with this by decomposing the selection of facts from a corpus autoregres- sively, conditioning the next iteration on previously selected facts. This allows us to use a pair- wise learning-to-rank loss.We validate our method on datasets of the TextGraphs 2019 and 2020 Shared Tasks for explanation regeneration. Existing work on this task either evaluates facts in isolation or artificially limits the possible chains of facts, thus limiting multi-hop inference. We demonstrate that our algorithm, when used with a pre-trained transformer model, outperforms the previous state-of-the-art in terms of precision, training time and inference efficiency. 1 Introduction The task of multi-hop explanation generation has recently received interest as it could be a stepping-stone towards general multi-hop inference over language. Multi-hop reasoning requires algorithms to combine multiple sources of evidence. This becomes increasingly hard when the number of required facts for an inference grows, because of the exploding number of combinations and phenomena such as semantic drift (Fried et al., 2015; Jansen, 2018). The WorldTree dataset was designed specifically for (>2)-fact inference (Jansen et al., 2018; Xie et al., 2020): it consists of elementary science exam questions that can be explained by an average of 6 facts from a complementary dataset of textual facts. The explanation regeneration task as in the TextGraphs Shared Tasks (Jansen and Ustalov, 2019; Jansen and Ustalov, 2020) asks participants to retrieve and rank relevant facts (given one of these natural language questions and its answer as input1) such that the top-ranked facts explain the answer to the question. An example is shown in the upper left part of fig. 1 (and more in appendix A). As each question-answer pair potentially has many supporting facts, evaluating all combinations of facts is computationally prohibitive. Previous work remedies this by computing scores for facts in isola- tion, or by severely limiting the number of combinations of facts (Das et al., 2019; Banerjee, 2019; Chia et al., 2019). The latter is done by considering combinations of few facts only and/or by reranking com- binations of only the top retrieved facts by a simpler method. Both approaches limit multi-hop reasoning as facts are not combined or too many facts are ignored by the simple first-stage retriever. In this paper we propose a method to retrieve facts that does build long chains of facts, while being ef- ficient. During training, a pre-trained neural language model encodes the question-answer pair as well as a randomly selected …

Content truncated to 500 words