Question Answering Infused Pre-training of General-Purpose Contextualized Representations

Findings of the Association for Computational Linguistics: ACL 2022, pages 711 - 728 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Question Answering Infused Pre-training of General-Purpose Contextualized Representations Robin Jia∗ University of Southern California robinjia@usc.edu Mike Lewis, Luke Zettlemoyer Facebook AI Research {mikelewis,lsz}@fb.com Abstract We propose a pre-training objective based on question answering (QA) for learning general- purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context. To this end, we train a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs. By encoding QA-relevant informa- tion, the bi-encoder’s token-level representa- tions are useful for non-QA downstream tasks without extensive (or in some cases, any) fine- tuning. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection on four datasets, few-shot named en- tity recognition on two datasets, and zero-shot sentiment analysis on three datasets. 1 Introduction While masked language models build contextual- ized word representations, they are pre-trained with losses that minimize distance to uncontextualized word embeddings (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). This objective yields a good initialization for downstream fine-tuning, but the pre-trained representations themselves are not optimized for being immediately useful without fine-tuning. In this paper, we introduce Question Answering Infused Pre-training (QUIP), a new pre- training loss based on question answering (QA) that depends much more directly on context. QUIP learns improved token-level representations that are useful in zero-shot and few-shot settings, where extensive fine-tuning is not possible. Our intuition for QUIP is that the contextualized representation for a phrase in a passage should con- tain enough information to identify all the questions ∗ Work done while a visiting researcher at Facebook AI Research. The Violin Concerto in D major, Op. 77, was composed by Johannes Brahms in 1878 and dedicated to his friend, the violinist Joseph Joachim. What did Brahms write in 1878? What was dedicated to Joachim? Who was friends with Joachim? Who played the violin? Who wrote the violin concerto? Figure 1: An overview of Question Answering Infused Pre-training. Our model independently creates vector representations (middle) for phrases in a passage (top) and for synthesized questions (bottom). Our objective encourages the vector for each phrase to have high simi- larity with the vectors for all questions it answers. that the phrase could answer in context. For exam- ple, in Figure 1, the representation for Johannes Brahms should be similar to the representation of all questions it can answer, such as “Who wrote the violin concerto?” We anticipate that optimiz- ing passage representations for QA should benefit many downstream tasks, as question-answer pairs have been used as broad-coverage meaning repre- sentations (He et al., 2015; Michael et al., 2018), and a wide range of NLP tasks can be cast as QA problems (Levy et al., 2017; …

Content truncated to 500 words