Contextualized Sparse Representations for Real-Time Open-Domain Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 912–919 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 912 Contextualized Sparse Representations for Real-Time Open-Domain Question Answering Jinhyuk Lee1 Minjoon Seo2,3 Hannaneh Hajishirzi2,4 Jaewoo Kang1 Korea University1 University of Washington2 Clova AI, NAVER3 Allen Institute for AI4 {jinhyuk lee,kangj}@korea.ac.kr {minjoon,hannaneh}@cs.washington.edu Abstract Open-domain question answering can be for- mulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accu- racy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embed- ding by augmenting it with a contextualized sparse representation (SPARC). Unlike pre- vious sparse vectors that are term-frequency- based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vec- tors in n-gram vocabulary space. By augment- ing the previous phrase retrieval model (Seo et al., 2019) with SPARC, we show 4%+ improvement in CuratedTREC and SQuAD- Open. Our CuratedTREC score is even bet- ter than the best known retrieve & read model with at least 45x faster inference speed.1 1 Introduction Open-domain question answering (QA) is the task of answering generic factoid questions by looking up a large knowledge source, typically unstructured text corpora such as Wikipedia, and finding the an- swer text segment (Chen et al., 2017). One widely adopted strategy to handle such large corpus is to use an efficient document (or paragraph) retrieval technique to obtain a few relevant documents, and then use an accurate (yet expensive) QA model to read the retrieved documents and find the an- swer (Chen et al., 2017; Wang et al., 2018; Das et al., 2019; Yang et al., 2019). More recently, an alternative approach formu- lates the task as an end-to-end phrase retrieval prob- lem by encoding and indexing every possible text span in a dense vector offline (Seo et al., 2018). The approach promises a massive speed advantage with 1Code available at https://github.com/jhyuklee/sparc. Overview Figure Passage Between 1991 and 2000, the total area of forest lost in the Amazon rose from 415,000 to 587,000 square kilometres Question How many square kilometres of the Amazon forest was lost by 1991? Sparse Representations of 415,000 tf-idf: amazon rose (1.00), . . . , 1991 (0.23), 2000 (0.19) Ours: 1991 (1.00), amazon (0.50), . . . , 2000 (0.17) Sparse Representations of 587,000 tf-idf: amazon rose (1.00), . . . , 1991 (0.23), 2000 (0.19) Ours: 2000 (1.00), amazon (0.53), . . . , 1991 (0.21) Figure 1: An example of sparse vectors given a con- text from SQuAD. While tf-idf has high weights on in- frequent n-grams, our contextualized sparse representa- tion (SPARC) focuses on sematically related n-grams. several orders of magnitude lower time complexity, but it performs poorly on entity-centric questions, often unable to disambiguate similar but different entities such as “1991” and “2001” in dense vec- tor space. To alleviate this issue, Seo et …

Content truncated to 500 words