Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4928–4949 July 9-14, 2023 ©2023 Association for Computational Linguistics Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments Yu Gu The Ohio State University gu.826@osu.edu Xiang Deng The Ohio State University deng.595@osu.edu Yu Su The Ohio State University su.809@osu.edu Abstract A key missing capacity of current language models (LMs) is grounding to real-world en- vironments. Most existing work for grounded language understanding uses LMs to directly generate plans that can be executed in the en- vironment to achieve the desired effects. It thereby casts the burden of ensuring grammati- cality, faithfulness, and controllability all on the LMs. We propose Pangu, a generic framework for grounded language understanding that capi- talizes on the discriminative ability of LMs in- stead of their generative ability. Pangu consists of a symbolic agent and a neural LM working in a concerted fashion: The agent explores the environment to incrementally construct valid plans, and the LM evaluates the plausibility of the candidate plans to guide the search process. A case study on the challenging problem of knowledge base question answering (KBQA), which features a massive environment, demon- strates the remarkable effectiveness and flexi- bility of Pangu: A BERT-base LM is sufficient for setting a new record on standard KBQA datasets, and larger LMs further bring substan- tial gains. Pangu also enables, for the first time, effective few-shot in-context learning for KBQA with large LMs such as Codex.1 1 Introduction Language models (LMs) such as BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), and Codex (Chen et al., 2021a) have demonstrated an extraordinary capacity in understanding and gener- ating both natural language (Minaee et al., 2021; Liang et al., 2022) and generic programs (e.g., Python) (Li et al., 2022; Jain et al., 2022; Austin et al., 2021). The recent release of ChatGPT and similar large LMs is elevating this paradigm to a new level. It seems to point us towards a future where natural language serves as a universal device, powered by LMs, for automated problem solving and interacting with the (computing) world. 1The Pangu library: OSU-NLP-Group/Pangu. Utterance Databases Knowledge Bases Tables Web Pages Apps Physical World Environments Plans Grammatical Faithful Symbolic Neural Scores Controllable Figure 1: A schematic illustration of the proposed frame- work, Pangu, where a symbolic agent interacts with the target environment to propose candidate plans, and a neural LM evaluates the plausibility of each plan. The agent searches the environment to incrementally con- struct the plans, and the LM guides the search process. However, a key missing piece in realizing this future is the connection between LMs and real- world environments, including both digital envi- ronments (e.g., databases, knowledge bases, Excel spreadsheets, software, websites, among others) and physical environments (e.g., instruction fol- lowing robots (Shridhar et al., 2020; Ahn et al., 2022)). Such environments are where many real problems lie. For example, a biologist may need to find all the species …

Content truncated to 500 words