Teaching Machine Comprehension with Compositional Explanations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1599–1615 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1599 Teaching Machine Comprehension with Compositional Explanations Qinyuan Ye1 Xiao Huang1 Elizabeth Boschee2 Xiang Ren1,2 1Department of Computer Science, University of Southern California 2Information Science Institute, University of Southern California {qinyuany, huan183}@usc.edu, boschee@isi.edu, xiangren@usc.edu Abstract Advances in machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated examples in the form of (question, paragraph, answer) triples. In contrast, humans are typically able to gener- alize with only a few examples, relying on deeper underlying world knowledge, linguis- tic sophistication, and/or simply superior de- ductive powers. In this paper, we focus on “teaching” machines reading comprehension, using a small number of semi-structured ex- planations that explicitly inform machines why answer spans are correct. We extract struc- tured variables and rules from explanations and compose neural module teachers that annotate instances for training downstream MRC mod- els. We use learnable neural modules and soft logic to handle linguistic variation and over- come sparse coverage; the modules are jointly optimized with the MRC model to improve fi- nal performance. On the SQuAD dataset, our proposed method achieves 70.14% F1 score with supervision from 26 explanations, compa- rable to plain supervised learning using 1,100 labeled instances, yielding a 12x speed up1. 1 Introduction Recent advances in neural sequence learning and pre-trained language models yield strong (human- level) performance on several reading comprehen- sion datasets (Lan et al., 2020; Raffel et al., 2019). However, state-of-the-art results mainly rely on large-scale annotated corpora, which are often time- consuming and costly to collect (Rajpurkar et al., 2016). This often leads to a large gap between methods in the research settings and practical use cases, as large amounts of annotated data rarely ex- ist for a new task or a low-resource domain (Linzen, 2020). To reduce this dependency on annotation efforts, we seek to improve the efficiency in obtain- ing and applying human supervision. 1Our code and data can be found at https://github. com/INK-USC/mrc-explanation. Reference Instance Q: When was Queen Victoria’s funeral ✿✿✿✿ held? C: Her funeral was ✿✿✿✿ held on Saturday, 2 February, in St George’s Chapel, Windsor Castle, and after two days of lying-in-state ... A: Saturday, 2 February Semi-structured Explanation X is “funeral”. Y is “✿✿✿✿ held”. In the question X is within 4 words after “when was” and Y is directly after X. “on” is directly before the answer. Y is within 2 words before the answer. X is within 3 words left of Y. The question starts with “when”, so the answer should be a date. Strictly-matched Instance Q: When was independence ✿✿✿✿✿✿✿✿ declared? C: ... Independence was ✿✿✿✿✿✿✿✿ declared on 24 September 1973. A: 24 September 1973 Softly-matched Instance Q: When was Brazelton ✿✿✿✿✿ killed? C: ... Brazelton was eventually tracked down and ✿✿✿✿✿ killed on Mon- day August 19, 1878, in a mesquite bosque ... A: Monday August 19, 1878 (Confidence z = 93.75%) Note: X is …

Content truncated to 500 words