Synthetic Question Value Estimation for Domain Adaptation of Question Answering

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1340 - 1351 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Synthetic Question Value Estimation for Domain Adaptation of Question Answering Xiang Yue1, Ziyu Yao2, and Huan Sun1 1The Ohio State University 2George Mason University {yue.149, sun.397}@osu.edu ziyuyao@gmu.edu Abstract Synthesizing QA pairs with a question gener- ator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models. Since syn- thetic questions are often noisy in practice, ex- isting work adapts scores from a pretrained QA (or QG) model as criteria to select high- quality questions. However, these scores do not directly serve the ultimate goal of improv- ing QA performance on the target domain. In this paper, we introduce a novel idea of train- ing a question value estimator (QVE) that di- rectly estimates the usefulness of synthetic questions for improving the target-domain QA performance. By conducting comprehensive experiments, we show that the synthetic ques- tions selected by QVE can help achieve bet- ter target-domain QA performance, in compar- ison with existing techniques. We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable per- formance to the fully-supervised baselines.1 1 Introduction Question answering (QA) systems based on pre- trained language models such as BERT (Devlin et al., 2019) have recently achieved promising performance in machine reading comprehension. However, neural QA systems trained on one do- main may not generalize well to another, leaving it challenging to deploy such systems on new do- mains that lack large-scale QA training data2. In this paper, we are interested in semi-supervised domain adaptation: we aim to build a target QA model with source-domain data and a small number of target-domain annotated QA pairs. 1Our source code is available at: https://github. com/xiangyue9607/QVE 2Large-scale training data are typically 60-100K in size. Existing: Repurposing QA/QG models for selection QA Model 1) Roundtrip Consistency (RTC) QG Model Context Answer Generated Question extracted answer matches? 2) LM Score Filtering QG Model QVE: learn from feedback on target annotations Log Likelihood Q1: -0.01 Q2: -0.05 Q3: -0.23 Q4: -2.31 Question Value Estimator QG Model QA Model Target Annotations Eval Reward: Target Performance Gain Select Useful QAs Generated Question Question: What was the nickname of the woman who allegedly provided call girls for prostitution? RTC LM QVE Generated Questions Context: ...Turbiville called herself the "Heidi Fleiss of Houston," referring to a woman who was dubbed the "Hollywood Madam" for providing call girls to... Question: Who is the founder of CNN? Context: ...CNN founder Ted says he is "encouraged" by the results of last week's election... ✘ Question: Who is recommended for Supreme Court? Context: ...The nomination of Elena Kagan to fill the seat of retiring Supreme Court Justice Stevens... ✓ ✘ ✓ ✘ ✓ ✘ ✓ Simple fact Mis- match High- quality Type ✓ : selected : filtered ✘ ✘ Figure 1: …

Content truncated to 500 words