ISAAQ - Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5469–5479, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5469 ISAAQ - Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention Jose Manuel Gomez-Perez Expert.AI Research Lab 10 Prof. Waksman 28036 Madrid, Spain jmgomez@expert.ai Raul Ortega Expert.AI Research Lab 10 Prof. Waksman 28036 Madrid, Spain rortega@expert.ai Abstract Textbook Question Answering is a complex task in the intersection of Machine Compre- hension and Visual Question Answering that requires reasoning with multimodal informa- tion from text and diagrams. For the first time, this paper taps on the potential of trans- former language models and bottom-up and top-down attention to tackle the language and visual understanding challenges this task en- tails. Rather than training a language-visual transformer from scratch we rely on pre- trained transformers, fine-tuning and ensem- bling. We add bottom-up and top-down atten- tion to identify regions of interest correspond- ing to diagram constituents and their relation- ships, improving the selection of relevant vi- sual information for each question and answer options. Our system ISAAQ reports unprece- dented success in all TQA question types, with accuracies of 81.36%, 71.11% and 55.12% on true/false, text-only and diagram multiple choice questions. ISAAQ also demonstrates its broad applicability, obtaining state-of-the- art results in other demanding datasets. 1 Introduction Within NLP, machine understanding of textbooks is one of the grand AI challenges. As originally put by (Reddy, 1988): ”Reading a chapter in a college freshman text (say physics or accounting) and an- swering the questions at the end of the chapter is a hard (AI) problem that requires advances in vision, language, problem-solving, and learning theory.”. Towards such goals, the Textbook Question An- swering (TQA) dataset presented in (Kembhavi et al., 2017) offers an excellent testbed. Drawn from middle school science curricula, it describes fairly complex phenomena through a combination of text and diagrams. Answering questions may therefore involve text, diagrams or both, and re- quire information from multiple sentences and/or diagrams in long textbook lessons. Another characteristic of the TQA dataset that makes it rather unique and challenging is that questions often involve reasoning beyond methods based on co-occurrence analysis or simple look-up. TQA requires parsing information from different sentences, dealing with qualitative and quantitative information (high frequency vs. 20,000Hz), and relating text or visual information with the ques- tion. Solving the TQA task also requires dealing with language about negation, conjunction, polarity or commonsense. On the visual side, TQA is rich with diagrams that describe potentially complex concepts, such as photosynthesis, the trophic chain, and the cycle of water, which are hard to represent as a single natural image. Quite on the contrary, diagrams contain simpler constituents and relation- ships between them, whose semantics needs to be captured in order to answer the questions. Despite recent work, overall progress in the TQA dataset has been rather limited until now, suggest- ing that language and diagram understanding chal- lenges like the ones above-mentioned are still far from solved. In …

Content truncated to 500 words