QuASE: Question-Answer Driven Sentence Encoding

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8743–8758 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8743 QUASE: Question-Answer Driven Sentence Encoding Hangfeng He† Qiang Ning‡∗ Dan Roth† † University of Pennsylvania ‡ Allen Institute for AI hangfeng,danroth@seas.upenn.edu qiangn@allenai.org Abstract Question-answering (QA) data often encodes essential information in many facets. This pa- per studies a natural question: Can we get su- pervision from QA data for other tasks (typ- ically, non-QA ones)? For example, can we use QAMR (Michael et al., 2017) to improve named entity recognition? We suggest that simply further pre-training BERT is often not the best option, and propose the question- answer driven sentence encoding (QUASE) framework. QUASE learns representations from QA data, using BERT or other state-of- the-art contextual language models. In particu- lar, we observe the need to distinguish between two types of sentence encodings, depending on whether the target task is a single- or multi- sentence input; in both cases, the resulting en- coding is shown to be an easy-to-use plugin for many downstream tasks. This work may point out an alternative way to supervise NLP tasks.1 1 Introduction It is labor-intensive to acquire human annotations for NLP tasks which require research expertise. For instance, one needs to know thousands of semantic frames in order to provide semantic role labelings (SRL) (Palmer et al., 2010). It is thus an important research direction to investigate how to get supervi- sion signals from indirect data and improve one’s target task. This paper studies the case of learning from question-answering (QA) data for other tasks (typically not QA). We choose QA because (1) a growing interest of QA has led to many large-scale QA datasets available to the community; (2) a QA task often requires comprehensive understanding of language and may encode rich information that ∗Part of this work was done while the author was at the University of Illinois at Urbana-Champaign. 1Our code and online demo are publicly available at https://github.com/CogComp/QuASE. is useful for other tasks; (3) it is much easier to answer questions relative to a sentence than to an- notate linguistics phenomena in it, making this a plausible supervision signal (Roth, 2017). There has been work showing that QA data for task A can help another QA task T , conceptually by further pre-training the same model on A (an of- ten larger) before training on T (a smaller) (Talmor and Berant, 2019; Sun et al., 2019). However, it remains unclear how to use these QA data when the target task does not share the same model as the QA task, which is often the case when the target task is not QA. For instance, QA-SRL (He et al., 2015), which uses QA pairs to represent those predicate- argument structures in SRL, should be intuitively helpful for SRL parsing, but the significant differ- ence in their surface forms prevents us from using the same model in both tasks. The success of modern language modeling tech- niques, …

Content truncated to 500 words