Locke’s Holiday: Belief Bias in Machine Reading

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8240–8245 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8240 Locke’s Holiday: Belief Bias in Machine Reading Anders Søgaard∗ CoAStaL, University of Copenhagen soegaard@di.ku.dk Abstract I highlight a simple failure mode of state-of- the-art machine reading systems: when con- texts do not align with commonly shared be- liefs. For example, machine reading systems fail to answer What did Elizabeth want? cor- rectly in the context of ’My kingdom for a cough drop, cried Queen Elizabeth.’ Biased by co-occurrence statistics in the training data of pretrained language models, systems pre- dict my kingdom, rather than a cough drop. I argue such biases are analogous to human belief biases and present a carefully designed challenge dataset for English machine reading, called AUTO-LOCKE, to quantify such effects. Evaluations of machine reading systems on AUTO-LOCKE show the pervasiveness of be- lief bias in machine reading. 1 Introduction Reading comprehension models are biased in many ways: they often expect lexical overlap between answer and question (Schlegel et al., 2020), expect the answers to occur in specific positions (Jia and Liang, 2017), or expect answers to be named enti- ties (Rondeau and Hazen, 2018). This paper con- siders belief bias (Sternberg and Leighton, 2004; Anderson and Hartzler, 2014) in the context of ma- chine reading based on language models. Belief bias is a type of cognitive bias, defined in psy- chology as the tendency to evaluate a statement based on prior beliefs rather than its logical strength (Evans et al., 1983). In Figure 1, the answer (’Ger- many’) follows straightforwardly from the context (without inference), but machine reading models nevertheless err, presumably because the prediction (’Malaysia’) aligns better with associations learned by language models. In another example, models are presented with the following context: ’Wash- ington is a number. Boston is a city.’ In this context, the models evaluated below were unable to answer ∗Each author wishes the others had contributed more. Context Indonesia is the Germany of the Asean. So then, Malaysia is the France. Question What country is Indonesia similar to? Answer Germany Prediction Malaysia Figure 1: Real-life example from Twitter (through http://metropho.rs). Both ELMo-BiDAF and NAQANet, for example, err on this simple machine reading problem in spite of the very short context (and thereby very limited set of potential answers). See Ta- ble 2 for hand-tailored examples used as templates in AUTO-LOCKE and more real-life examples. even the simple question of What is Washington? Instead of answering ’a number’, they consistently answered ’a city’. Contributions Below I present a somewhat hap- hazard evaluation of four common machine reading systems – BiDAF, ELMo-BiDAF, TransformerQA, and NAQANet – on a new benchmark called AUTO- LOCKE, consisting of variations across a manually constructed collection of examples that are, at the same time, trivially easy for humans (as shown below through crowd-sourcing), yet violate world knowledge or commonly held beliefs, e.g., that Washington is a city. I show that a) across the board, machine …

Content truncated to 500 words