Benefits of Intermediate Annotations in Reading Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5627–5634 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5627 Benefits of Intermediate Annotations in Reading Comprehension Dheeru Dua University of California, Irvine, CA, USA ddua@uci.edu Sameer Singh University of California, Irvine, CA, USA sameer@uci.edu Matt Gardner Allen Institute for Artificial Intelligence, Irvine, CA, USA mattg@allenai.org Abstract Complex, compositional reading comprehen- sion datasets require performing latent sequen- tial decisions that are learned via supervision from the final answer. A large combinatorial space of possible decision paths that result in the same answer, compounded by the lack of intermediate supervision to help choose the right path, makes the learning particularly hard for this task. In this work, we study the bene- fits of collecting intermediate reasoning super- vision along with the answer during data col- lection. We find that these intermediate anno- tations can provide two-fold benefits. First, we observe that for any collection budget, spend- ing a fraction of it on intermediate annotations results in improved model performance, for two complex compositional datasets: DROP and Quoref. Second, these annotations encour- age the model to learn the correct latent reason- ing steps, helping combat some of the biases introduced during the data collection process. 1 Introduction Recently many reading comprehension datasets requiring complex and compositional reason- ing over text have been introduced, including HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019), Quoref (Dasigi et al., 2019), and ROPES (Lin et al., 2019). However, models trained on these datasets (Hu et al., 2019; Andor et al., 2019) only have the final answer as supervision, leaving the model guessing at the correct latent rea- soning. Figure 1 shows an example from DROP, which requires first locating various operands (i.e. relevant spans) in the text and then performing fil- ter and count operations over them to get the final answer “3”. However, the correct answer can also be obtained by extracting the span “3” from the passage, or by adding or subtracting various num- bers in the passage. The lack of intermediate hints makes learning challenging and can lead the model Question: How many touchdown passes did Cutler throw in the second half? Answer: 3 .....In the third quarter, the Vikes started to rally with run- ning back Adrian Peterson’s 1-yard touchdown run (with the extra point attempt blocked). The Bears increased their lead over the Vikings with Cutler’s 3-yard TD pass to tight end Desmond Clark. The Vikings then closed out the quarter with quarterback Brett Favre firing a 6-yard TD pass to tight end Visanthe Shiancoe. An exciting .... with kicker Ryan Longwell’s 41-yard field goal, along with Adrian Peterson’s second 1-yard TD run. The Bears then responded with Cutler firing a 20-yard TD pass to wide receiver Earl Bennett. The Vikings then completed the remarkable comeback with Favre finding wide receiver Sidney Rice on a 6-yard TD pass on 4th-and-goal with 15 seconds left in regulation. The Bears then took a knee to force …

Content truncated to 500 words