Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension

Proceedings of NAACL-HLT 2019, pages 2644–2655 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2644 Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension Yichong Xu1∗, Xiaodong Liu2, Yelong Shen2, Jingjing Liu2 and Jianfeng Gao2 1 Carnegie Mellon University 2 Microsoft Research yichongx@cs.cmu.edu {xiaodl, yeshen, jingjl,jfgao}@microsoft.com Abstract We propose a multi-task learning framework to learn a joint Machine Reading Compre- hension (MRC) model that can be applied to a wide range of MRC tasks in different do- mains. Inspired by recent ideas of data se- lection in machine translation, we develop a novel sample re-weighting scheme to assign sample-specific weights to the loss. Empiri- cal study shows that our approach can be ap- plied to many existing MRC models. Com- bined with contextual representations from pre-trained language models (such as ELMo), we achieve new state-of-the-art results on a set of MRC benchmark datasets. We re- lease our code at https://github.com/ xycforgithub/MultiTask-MRC. 1 Introduction Machine Reading Comprehension (MRC) has gained growing interest in the research commu- nity (Rajpurkar et al., 2016; Yu et al., 2018). In an MRC task, the machine reads a text passage and a question, and generates (or selects) an answer based on the passage. This requires the machine to possess strong comprehension, inference and reasoning capabilities. Over the past few years, there has been much progress in building end-to- end neural network models (Seo et al., 2016) for MRC. However, most public MRC datasets (e.g., SQuAD, MS MARCO, TriviaQA) are typically small (less than 100K) compared to the model size (such as SAN (Liu et al., 2018c,b) with around 10M parameters). To prevent over-fitting, recently there have been some studies on using pre-trained word embeddings (Pennington et al., 2014) and contextual embeddings in the MRC model train- ing, as well as back-translation approaches (Yu et al., 2018) for data augmentation. ∗Most of this work was performed when the author was interning at Microsoft. Multi-task learning (Caruana, 1997) is a widely studied area in machine learning, aiming at bet- ter model generalization by combining training datasets from multiple tasks. In this work, we explore a multi-task learning (MTL) framework to enable the training of one universal model across different MRC tasks for better generaliza- tion. Intuitively, this multi-task MRC model can be viewed as an implicit data augmentation tech- nique, which can improve generalization on the target task by leveraging training data from aux- iliary tasks. We observe that merely adding more tasks can- not provide much improvement on the target task. Thus, we propose two MTL training algorithms to improve the performance. The first method sim- ply adopts a sampling scheme, which randomly selects training data from the auxiliary tasks con- trolled by a ratio hyperparameter; The second al- gorithm incorporates recent ideas of data selection in machine translation (van der Wees et al., 2017). It learns the sample weights from the auxiliary tasks automatically through language models. Prior to this work, many studies have used up- stream datasets to augment the …

Content truncated to 500 words