Unsupervised Domain Adaptation of Language Models for Reading Comprehension

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5392–5399 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5392 Unsupervised Domain Adaptation of Language Models for Reading Comprehension Kosuke Nishida, Kyosuke Nishida, Itsumi Saito, Hisako Asano, Junji Tomita NTT Media Intelligence Laboratories, NTT Corporation 1-1 Hikarinooka Yokosuka-Shi, Kanagawa, Japan kosuke.nishida.ap@hco.ntt.co.jp Abstract This study tackles unsupervised domain adaptation of reading comprehension (UDARC). Reading comprehension (RC) is a task to learn the capability for question answering with textual sources. State-of-the-art models on RC still do not have general linguistic intelligence; i.e., their accuracy worsens for out-domain datasets that are not used in the training. We hypothesize that this discrepancy is caused by a lack of the language modeling (LM) capability for the out-domain. The UDARC task allows models to use supervised RC training data in the source domain and only unlabeled passages in the target domain. To solve the UDARC problem, we provide two domain adaptation models. The first one learns the out-domain LM and in-domain RC task sequentially. The second one is the proposed model that uses a multi-task learning approach of LM and RC. The models can retain both the RC capability acquired from the supervised data in the source domain and the LM capability from the unlabeled data in the target domain. We evaluated the models on UDARC with five datasets in different domains. The models outperformed the model without domain adaptation. In particular, the proposed model yielded an improvement of 4.3/4.2 points in EM/F1 in an unseen biomedical domain. Keywords: Reading Comprehension, Domain Adaptation, Unsupervised Learning 1. Introduction Reading comprehension (RC) is a task to acquire a capabil- ity of understanding natural language for question answer- ing with textual sources. It has seen significant progress since the release of numerous datasets such as SQuAD (Rajpurkar et al., 2016) and the rise of the deep neural models such as BiDAF (Seo et al., 2017). Recently, fine- tuning of pre-trained language models (LM) such as BERT (Devlin et al., 2019) has achieved state-of-the-art perfor- mance in many NLP tasks including RC. However, such state-of-the-art models still do not have gen- eral linguistic intelligence; e.g., their accuracy is sensitively affected by the difference in the distribution between the training and evaluation datasets, such as in the domains of textual sources. This discrepancy becomes an issue in a real-world scenario. For instance, a business intending to introduce an RC application for a service must create tens of thousands of (passage, query, answer) tuples per domain of the service (Yogatama et al., 2019). However, such a large annotation to create training data costs much money and takes up a lot of time. When the domain entails personal information or expert knowledge, even crowd- sourcing can not be used to create the training data. This study tackles a task, called Unsupervised Domain Adaptation of Reading Comprehension (UDARC). Ta- ble 1 shows the task setting of UDARC. The model can use the (passage, query, answer) tuples …

Content truncated to 500 words