Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2553–2566, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2553 Revealing the Importance of Semantic Retrieval for Machine Reading at Scale Yixin Nie Songhe Wang UNC Chapel HillMohit Bansal {yixin1, songhe17, mbansal}@cs.unc.edu Abstract Machine Reading at Scale (MRS) is a chal- lenging task in which a system is given an input query and is asked to produce a pre- cise output by “reading” information from a large knowledge base. The task has gained popularity with its natural combination of in- formation retrieval (IR) and machine compre- hension (MC). Advancements in representa- tion learning have led to separated progress in both IR and MC; however, very few studies have examined the relationship and combined design of retrieval and comprehension at dif- ferent levels of granularity, for development of MRS systems. In this work, we give gen- eral guidelines on system design for MRS by proposing a simple yet effective pipeline sys- tem with special consideration on hierarchical semantic retrieval at both paragraph and sen- tence level, and their potential effects on the downstream task. The system is evaluated on both fact verification and open-domain multi- hop QA, achieving state-of-the-art results on the leaderboard test sets of both FEVER and HOTPOTQA. To further demonstrate the im- portance of semantic retrieval, we present ab- lation and analysis studies to quantify the con- tribution of neural retrieval modules at both paragraph-level and sentence-level, and illus- trate that intermediate semantic retrieval mod- ules are vital for not only effectively filtering upstream information and thus saving down- stream computation, but also for shaping up- stream data distribution and providing better data for downstream modeling.1 1 Introduction Extracting external textual knowledge for machine comprehensive systems has long been an impor- tant yet challenging problem. Success requires 1Code/data made publicly available at: https:// github.com/easonnie/semanticRetrievalMRS not only precise retrieval of the relevant informa- tion sparsely restored in a large knowledge source but also a deep understanding of both the selected knowledge and the input query to give the corre- sponding output. Initiated by Chen et al. (2017), the task was termed as Machine Reading at Scale (MRS), seeking to provide a challenging situation where machines are required to do both semantic retrieval and comprehension at different levels of granularity for the final downstream task. Progress on MRS has been made by improv- ing individual IR or comprehension sub-modules with recent advancements on representative learn- ing (Peters et al., 2018; Radford et al., 2018; De- vlin et al., 2018). However, partially due to the lack of annotated data for intermediate retrieval in an MRS setting, the evaluations were done mainly on the final downstream task and with much less consideration on the intermediate retrieval perfor- mance. This led to the convention that upstream retrieval modules mostly focus on getting better coverage of the downstream information such that the upper-bound of the downstream score can …

Content truncated to 500 words