Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2346–2357 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2346 Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension An Yang1*, Quan Wang2, Jing Liu2, Kai Liu2, Yajuan Lyu2, Hua Wu2†, Qiaoqiao She2 and Sujian Li1† 1Key Laboratory of Computational Linguistics, Peking University, MOE, China 2Baidu Inc., Beijing, China {yangan, lisujian}@pku.edu.cn {wangquan05, liujing46, liukai20, lvyajuan, wu hua, sheqiaoqiao}@baidu.com Abstract Machine reading comprehension (MRC) is a crucial and challenging task in NLP. Recently, pre-trained language models (LMs), especially BERT, have achieved remarkable success, pre- senting new state-of-the-art results in MRC. In this work, we investigate the potential of lever- aging external knowledge bases (KBs) to fur- ther improve BERT for MRC. We introduce KT-NET, which employs an attention mech- anism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context- and knowledge- aware predictions. We believe this would com- bine the merits of both deep LMs and curated KBs towards better MRC. Experimental re- sults indicate that KT-NET offers significant and consistent improvements over BERT, out- performing competitive baselines on ReCoRD and SQuAD1.1 benchmarks. Notably, it ranks the 1st place on the ReCoRD leaderboard, and is also the best single model on the SQuAD1.1 leaderboard at the time of submission (March 4th, 2019).1 1 Introduction Machine reading comprehension (MRC), which requires machines to comprehend text and answer questions about it, is a crucial task in natural lan- guage processing. With the development of deep learning and the increasing availability of datasets (Rajpurkar et al., 2016, 2018; Nguyen et al., 2016; Joshi et al., 2017), MRC has achieved remarkable advancements in the last few years. Recently language model (LM) pre-training has caused a stir in the MRC community. These LMs *This work was done while the first author was an intern at Baidu Inc. †Co-corresponding authors: Hua Wu and Sujian Li. 1Our code will be available at http://github. com/paddlepaddle/models/tree/develop/ PaddleNLP/Research/ACL2019-KTNET Passage: The US government has extended its review into whether trade sanctions against Sudan should be repealed. [...] Sudan is committed to the full implementation of UN Security Council resolutions on North Korea. [...] Sudan’s past support for North Korea could present an obstacle [...] Question: Sudan remains a XXX-designated state sponsor of terror and is one of six countries subject to the Trump administration’s ban. Original BERT prediction: UN Security Council Prediction with background knowledge: US Background knowledge: NELL: (Donald Trump, person-leads-organization, US) WordNet: (government, same-synset-with, administration) WordNet: (sanctions, common-hypernym-with, ban) Figure 1: An example from ReCoRD, with answer can- didates marked (underlined) in the passage. The vanilla BERT model fails to predict the correct answer. But it succeeds after integrating background knowledge col- lected from WordNet and NELL. are pre-trained on unlabeled text and then applied to MRC, in either a feature-based (Peters et al., 2018a) or a fine-tuning (Radford et al., 2018) man- ner, both offering substantial performance boosts. Among different pre-training …

Content truncated to 500 words