No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4141–4150 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4141 No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension Xuguang Wang1 Linjun Shou1 Ming Gong1 Nan Duan2 Daxin Jiang1‡ 1Microsoft STCA NLP Group, Beijing, China 2Microsoft Research Asia, Beijing, China {xugwang,lisho,migon,nanduan,djiang}@microsoft.com Abstract The Natural Questions (NQ) benchmark set brings new challenges to Machine Reading Comprehension: the answers are not only at different levels of granularity (long and short), but also of richer types (including no-answer, yes/no, single-span and multi-span). In this paper, we target at this challenge and handle all answer types systematically. In particu- lar, we propose a novel approach called Re- flection Net which leverages a two-step train- ing procedure to identify the no-answer and wrong-answer cases. Extensive experiments are conducted to verify the effectiveness of our approach. At the time of paper writing (May. 20, 2020), our approach achieved the top 1 on both long and short answer leader- board∗, with F1 scores of 77.2 and 64.1, re- spectively. 1 Introduction Deep neural network models, such as (Cui et al., 2017; Chen et al., 2017; Clark and Gardner, 2018; Wang et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019), have greatly advanced the state-of-the-arts of machine read- ing comprehension (MRC). Natural Questions (NQ) (Kwiatkowski et al., 2019) is a new Question Answering benchmark released by Google, which brings new challenges to the MRC area. One chal- lenge is that the answers are provided at two-level granularity, i.e., long answer (e.g., a paragraph in the document) and short answer (e.g., an entity or entities in a paragraph). Therefore, the task requires the models to search for answers at both document level and passage level. Moreover, there are richer answer types in the NQ task. In addition to indi- cating textual answer spans (long and short), the ‡Corresponding author. ∗https://ai.google.com/research/ NaturalQuestions/leaderboard (a) Question: who made it to stage 3 in american ninja warrior sea- son 9 Wikipedia Page: American Ninja Warrior (season 9) Long Answer: Results: Joe Moravsky (3:34.34), Najee Richardson (3:39:71) and Sean Bryan finished to go into Stage 3. Short Answer: Joe Moravsky, Najee Richardson, Sean Bryan (b) Question: why does queen Elizabeth sign her name Elizabeth r Wikipedia Page: Royal sign-manual Long Answer: The royal sign-manual usually consists of the sovereign’s regnal name (without number, if other- wise used), followed by the letter R for Rex (King) or Regina (Queen). Thus, the signs-manual of both Elizabeth I and Elizabeth II read Elizabeth R ... Short Answer: NULL (c) Question: is an end of terraced house semi detached Wikipedia Page: Terraced house Long Answer: In the 21st century, Montr´eal has continued to build row houses at a high rate, with 62% of housing starts in the metropolitan area being apartment or row units.[10]Apartment complexes, high-rises, and semi-detached homes are less popular in Montr´eal when compared to large Canadian cities ... Short Answer: …

Content truncated to 500 words