Long-Tailed Question Answering in an Open World

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6362–6382 July 9-14, 2023 ©2023 Association for Computational Linguistics Long-Tailed Question Answering in an Open World Yi Dai1∗†, Hao Lang2†‡, Yinhe Zheng2, Fei Huang2, Yongbin Li2‡ 1 Department of Computer Science and Technology, Tsinghua University 2 Alibaba Group {hao.lang, f.huang, shuide.lyb}@alibaba-inc.com, dai-y21@mails.tsinghua.edu.cn, zhengyinhe1@163.com Abstract Real-world data often have an open long-tailed distribution, and building a unified QA model supporting various tasks is vital for practical QA applications. However, it is non-trivial to extend previous QA approaches since they ei- ther require access to seen tasks of adequate samples or do not explicitly model samples from unseen tasks. In this paper, we define Open Long-Tailed QA (OLTQA) as learning from long-tailed distributed data and optimiz- ing performance over seen and unseen QA tasks. We propose an OLTQA model that en- courages knowledge sharing between head, tail and unseen tasks, and explicitly mines knowl- edge from a large pre-trained language model (LM). Specifically, we organize our model through a pool of fine-grained components and dynamically combine these components for an input to facilitate knowledge sharing. A retrieve-then-rerank frame is further intro- duced to select in-context examples, which guild the LM to generate text that express knowledge for QA tasks. Moreover, a two- stage training approach is introduced to pre- train the framework by knowledge distillation (KD) from the LM and then jointly train the frame and a QA model through an adaptive mutual KD method. On a large-scale OLTQA dataset we curate from 43 existing QA datasets, our model consistently outperforms the state- of-the-art. We release the code and data at https://github.com/AlibabaResearch/ DAMO-ConvAI/tree/main/oltqa. 1 Introduction Real-world data often have a long-tailed and open- ended distribution (Liu et al., 2019b). As a cor- nerstone for AI applications (Yang et al., 2019), Question Answering (QA) is widely investigated to tackle various QA tasks involving diverse formats ∗ Work done while the author was interning at Alibaba. † Equal contribution. ‡ Corresponding author. and domains (Khashabi et al., 2020b; Zhong et al., 2022a). The frequency distribution of QA tasks in our daily life is long-tailed (Reed, 2001), with a few head tasks of adequate samples and many more tail tasks of limited samples, and we continu- ously encounter new tasks that are not seen during training in an open world. We formally study Open Long-Tailed QA (OLTQA) emerging in natural data settings. A practical QA system shall learn from long-tailed distributed data, i.e., a few head tasks and many tail tasks, and it is expected to perform well over a balanced test set which include head, tail, and unseen tasks. OLTQA must handle not only few-shot learning for tail tasks in the closed world (Shu et al., 2017), but also zero-shot learning for unseen tasks in an open world (Scheirer et al., 2012) with one unified model. A major challenge for OLTQA is the lack of knowledge required for the language understanding and reasoning abilities of QA tasks, …

Content truncated to 500 words