Textbook Question Answering with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3568–3584 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3568 Textbook Question Answering with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension Daesik Kim1,2,∗ Seonhoon Kim1,3,∗ Nojun Kwak1,† 1Seoul National University 2V.DO Inc. 3Search&Clova, Naver Corp. {daesik.kim|nojunk}@snu.ac.kr seonhoon.kim@navercorp.com Abstract In this work, we introduce a novel algorithm for solving the textbook question answering (TQA) task which describes more realistic QA problems compared to other recent tasks. We mainly focus on two related issues with anal- ysis of the TQA dataset. First, solving the TQA problems requires to comprehend multi- modal contexts in complicated input data. To tackle this issue of extracting knowledge fea- tures from long text lessons and merging them with visual features, we establish a context graph from texts and images, and propose a new module f-GCN based on graph con- volutional networks (GCN). Second, scien- tific terms are not spread over the chapters and subjects are split in the TQA dataset. To overcome this so called ‘out-of-domain’ is- sue, before learning QA problems, we intro- duce a novel self-supervised open-set learn- ing process without any annotations. The ex- perimental results show that our model signifi- cantly outperforms prior state-of-the-art meth- ods. Moreover, ablation studies validate that both methods of incorporating f-GCN for ex- tracting knowledge from multi-modal contexts and our newly proposed self-supervised learn- ing process are effective for TQA problems. 1 Introduction In a decade, question answering (QA) has been one of the most promising achievements in the field of natural language processing (NLP). Fur- thermore, it has shown great potential to be ap- plied to real-world problems. In order to solve more realistic QA problems, input types in datasets have evolved into various combinations. Recently, Visual Question Answering (VQA) has drawn huge attractions as it is in the intersection * Equal contribution. † Corresponding author. This work was supported by Next-Generation Information Computing Development Program through the National Re- search Foundation of Korea (NRF-2017M3C4A7078547). Nucleic acid classification fuction of nucleic acid DNA stores genetic information in the cells of all living things. It contains the genetic code. This is the code that instructs cells how to make proteins. nucleotide RNA consists of just one chain of nucleotides. DNA consists of two chains. Nitrogen bases on the two chains of DNA form hydrogen bonds with each other. Hydrogen bonds are relatively weak bonds that form between a positively charged hydrogen atom in one molecule and a negatively charged atom in another molecule. Context Graph Questions nitrogen bases in dna include a) adenine. b) uracil. c) ribose. d) two of the above What is the term for connected sugar, phosphate group and protein? a) hydrogen bond b) deoxyribose c) nucleotide d) sugar-phosphate backbone Comprehend + Solve LESSON Training Set Validation Set Training Set Figure 1: Examples of the textbook question answering task and a brief concept of our work. In this figure, we can see lessons which contain long …

Content truncated to 500 words