Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5947–5952, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5947 Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension Daniel Andor, Luheng He, Kenton Lee, Emily Pitler Google Research {andor, luheng, kentonl, epitler}@google.com Abstract Reading comprehension models have been successfully applied to extractive text answers, but it is unclear how best to generalize these models to abstractive numerical answers. We enable a BERT-based reading comprehension model to perform lightweight numerical rea- soning. We augment the model with a prede- fined set of executable ‘programs’ which en- compass simple arithmetic as well as extrac- tion. Rather than having to learn to manip- ulate numbers directly, the model can pick a program and execute it. On the recent Discrete Reasoning Over Passages (DROP) dataset, de- signed to challenge reading comprehension models, we show a 33% absolute improvement by adding shallow programs. The model can learn to predict new operations when appropri- ate in a math word problem setting (Roy and Roth, 2015) with very few training examples. 1 Introduction End-to-end reading comprehension models have been increasingly successful at extractive ques- tion answering. For example, performance on the SQuAD 2.0 (Rajpurkar et al., 2018) benchmark has improved from 66.3 F1 to 89.51 in a single year. However, the Discrete Reasoning Over Pas- sages (DROP) (Dua et al., 2019) dataset demon- strates that as long as there is quantitative reason- ing involved, there are plenty of relatively straight- forward questions that current extractive QA sys- tems find difficult to answer. Other recent work has shown that even state-of-the-art neural models struggle with numerical operations and quantita- tive reasoning when trained in an end-to-end man- ner (Saxton et al., 2019; Ravichander et al., 2019). In other words, even BERT (Devlin et al., 2019) is not very good at doing simple calculations. 1https://rajpurkar.github.io/SQuAD-explorer/ How many more Chinese nationals are there than Euro- pean nationals? The city of Bangkok has a population of 8,280,925 ...the census showed that it is home to 81,570 Japanese and 55,893 Chinese nationals, as well as 117,071 expatriates from other Asian countries, 48,341 from Europe, 23,418 from the Americas,... NAQANet: −55893 Ours: Diff(55893, 48341) = 7552 Table 1: Example from the DROP development set. The correct answer is not explicitly stated in the pas- sage and instead must be computed. The NAQANet model2(Dua et al., 2019) predicts a negative number of people, whereas our model predicts that an operation Diff should be taken and identifies the two arguments. In this work, we extend an extractive QA system with numerical reasoning abilities. We do so by asking the neural network to synthe- size small programs that can be executed. The model picks among simple programs of the form Operation(args, ...), where the possible oper- ations include span extraction, answering yes or no, and arithmetic. For math operations, the argu- ments are pointers to numbers in …

Content truncated to 500 words