ScholarlyRead: A New Dataset for Scientific Article Reading Comprehension

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5498–5504 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5498 ScholarlyRead: A New Dataset for Scientific Article Reading Comprehension Tanik Saikh, Asif Ekbal, Pushpak Bhattacharyya Indian Institute of Technology Patna Bihta, Patna, Bihar-801106, India {tanik.srf17, asif, pb}@iitp.ac.in Abstract We present ScholarlyRead, span-of-word-based scholarly articles’ Reading Comprehension (RC) dataset with approximately 10K manually checked passage-question-answer instances. ScholarlyRead was constructed in semi-automatic way. We consider the articles from two popular journals of a reputed publishing house. Firstly, we generate questions from these articles in an automatic way. Generated questions are then manually checked by the human annotators. We propose a baseline model based on Bi-Directional Attention Flow (BiDAF) network that yields the F1 score of 37.31%. The framework would be useful for building Question-Answering (QA) systems on scientific articles. Keywords: Reading Comprehension, Data Preparation, Scholarly Articles 1. Introduction Resurgence of Artificial Intelligence (AI) and various deep learning techniques have solved many complicated tasks successfully. It is also possible to learn a machine to com- prehend a document. The machine will be able to yield answers to questions based on that particular document. This task is typically called as Machine Reading Compre- hension (MRC)/Reading Comprehension (RC) (Hermann et al., 2015). The accompanying document (that remains the information, essential for answering the questions) of- ten termed as Context. Comprehension is the capability to understand something. Learning machine to understand hu- man language documents is one of the most elusive and well established challenges in AI. Prior works and datasets on MRC mostly concern with the general domain datasets of news articles and/or elementary school-level storybooks and/or on the articles contained in Wikipedia. Our focus is to enable machine to understand technical materials those are contained in scientific articles. Reading technical and scientific texts involves a complex process. It is entirely different from reading general content. So, MRC on schol- arly articles is more challenging compared to the other do- mains. Human being generally put their level best intelli- gence when they write a research article. It is very difficult to decode the content of the articles even for the human be- ing when they do review or read for research purpose. Readers, while reading a research article, generally read the abstract and conclusion before delving into the details of the article. This is also true for the editors or associate edi- tors who go through the abstract of the article to get an idea about the quality of the paper as well as assigning appro- priate reviewers. In this paper we first create a dataset for MRC in research article, and then develop a deep learning based model as a baseline for further research. The under- lying principle is to develop an AI Assisted Peer Review System which will help the editors and reviewers by pro- viding the answers to some basic questions. The datasets for this kind of tasks typically contain document-question- answer triples. The …

Content truncated to 500 words