Few-shot In-context Learning on Knowledge Base Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6966–6980 July 9-14, 2023 ©2023 Association for Computational Linguistics Few-shot In-context Learning for Knowledge Base Question Answering ♠Tianle Li, ♠Xueguang Ma, ♠Alex Zhuang, ♥Yu Gu, ♥Yu Su, ♠,♣Wenhu Chen ♠University of Waterloo ♥The Ohio State University ♣Vector Institute, Toronto {t29li,x93ma,a5zhuang,wenhuchen}@uwaterloo.ca, {gu.826,su.809}@osu.edu Abstract Question answering over knowledge bases is considered a difficult problem due to the chal- lenge of generalizing to a wide variety of possible natural language questions. Addi- tionally, the heterogeneity of knowledge base schema items between different knowledge bases often necessitates specialized training for different knowledge base question-answering (KBQA) datasets. To handle questions over diverse KBQA datasets with a unified training- free framework, we propose KB-BINDER, which for the first time enables few-shot in- context learning over KBQA tasks. Firstly, KB- BINDER leverages large language models like Codex to generate logical forms as the draft for a specific question by imitating a few demon- strations. Secondly, KB-BINDER grounds on the knowledge base to bind the generated draft to an executable one with BM25 score matching. The experimental results on four public heterogeneous KBQA datasets show that KB-BINDER can achieve a strong per- formance with only a few in-context demon- strations. Especially on GraphQA and 3-hop MetaQA, KB-BINDER can even outperform the state-of-the-art trained models. On GrailQA and WebQSP, our model is also on par with other fully-trained models. We believe KB- BINDER can serve as an important baseline for future research. Our code is available at https://github.com/ltl3A87/KB-BINDER 1 Introduction Question answering over knowledge bases (KBQA) (Berant et al., 2013; Yih et al., 2015) has been a long-standing research problem in the AI community. It has attracted wide attention from the community with its significant role in making large-scale knowledge bases accessible to non-expert users (Wu et al., 2019; Lan et al., 2021; Gu et al., 2022). However, despite the fact that the increasing scale of knowledge bases can enable the retrieval with higher coverage on miscellaneous Exemplars Question: --- Logical Form: --- Question: --- Logical Form: Question: How many game expansions has john elliott released? Generated Drafts Entity Binder Relation Binder Candidates KB Answer: 1 LLM Execute Figure 1: Overview of KB-BINDER pipeline. There are two primary stages in our method: 1) Generate the drafts as preliminary logical forms; 2) Bind the drafts to the executable ones with entity and relation binders grounded on the knowledge base. The final answer can be obtained after the execution of the final candidates. topics, it poses a great challenge for suppliers with limited resources, who rely on the models trained on certain knowledge bases or benchmarks. Concretely, the difficulties primarily lie in the following aspects: 1) Data intensiveness: larger knowledge bases require ever larger quantities of annotated data to allow fine-tuned models to generalize well over them. (Yih et al., 2016; Talmor and Berant, 2018; Gu et al., 2020). 2) Dataset specificity: For relatively small-scale KBQA datasets, the fully-trained models tend to overfit to …

Content truncated to 500 words