Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 835–844 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension David Golub∗ Stanford University golubd@cs.stanford.edu Po-Sen Huang and Xiaodong He Microsoft Research {pshuang, xiaohe}@microsoft.com Li Deng† Citadel Securities, LLC l.deng@ieee.org Abstract We develop a technique for transfer learn- ing in machine comprehension (MC) us- ing a novel two-stage synthesis network (SynNet). Given a high-performing MC model in one domain, our technique aims to answer questions about documents in another domain, where we use no labeled data of question-answer pairs. Using the proposed SynNet with a pretrained model on the SQuAD dataset, we achieve an F1 measure of 46.6% on the challeng- ing NewsQA dataset, approaching perfor- mance of in-domain models (F1 measure of 50.0%) and outperforming the out-of- domain baseline by 7.6%, without use of provided annotations.1 1 Introduction Machine comprehension (MC), the ability to an- swer questions over a provided context paragraph, is a key task in natural language processing. The rise of high-quality, large-scale human-annotated datasets for this task (Rajpurkar et al., 2016; Trischler et al., 2016) has allowed for the train- ing of data-intensive but expressive models such as deep neural networks (Wang et al., 2016; Xiong et al., 2017; Seo et al., 2017). Moreover, these datasets have the attractive quality that the answer is a short snippet of text within the paragraph, which narrows the search space of possible answer spans. However, many of these models rely on large amounts of human-labeled data for training. Yet ∗Work performed while interning at Microsoft Research. † Work performed when the author was at Microsoft Re- search. 1Code will be available at https://github.com/ davidgolub/QuestionGeneration data collection is a time-consuming and expensive task. Moreover, direct application of a MC model trained on one domain to answer questions over paragraphs from another domain may suffer per- formance degradation. While understudied, the ability to transfer a MC model to multiple domains is of great practical im- portance. For instance, the ability to quickly use a MC model trained on Wikipedia to bootstrap a question-answering system over customer support manuals or news articles, where there is no labeled data, can unlock a great number of practical appli- cations. In this paper, we address this problem in MC through a two-stage synthesis network (SynNet). The SynNet generates synthetic question-answer pairs over paragraphs in a new domain that are then used in place of human-generated annotations to finetune a MC model trained on the original do- main. The idea of generating synthetic data to aug- ment insufficient training data has been explored before. For example, for the target task of trans- lation, Sennrich et al. (2016) present a method to generate synthetic translations given real sen- tences to refine an existing machine translation system. However, unlike machine translation, for tasks like MC, we need to synthesize both the question and answers given the context paragraph. More- over, while the question is a …

Content truncated to 500 words