John praised Mary because _he_? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4859–4871 August 1–6, 2021. ©2021 Association for Computational Linguistics 4859 John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs Yova Kementchedjhieva University of Copenhagen yova@di.ku.dk Mark Anderson Universidade da Coru˜na, CITIC Department of CS & IT m.anderson@udc.es Anders Søgaard University of Copenhagen soegaard@di.ku.dk Abstract Some interpersonal verbs can implicitly at- tribute causality to either their subject or their object and are therefore said to carry an im- plicit causality (IC) bias. Through this bias, causal links can be inferred from a narrative, aiding language comprehension. We inves- tigate whether pre-trained language models (PLMs) encode IC bias and use it at inference time. We find that to be the case, albeit to different degrees, for three distinct PLM ar- chitectures. However, causes do not always need to be implicit—when a cause is explicitly stated in a subordinate clause, an incongruent IC bias associated with the verb in the main clause leads to a delay in human processing. We hypothesize that the temporary challenge humans face in integrating the two contradict- ing signals, one from the lexical semantics of the verb, one from the sentence-level seman- tics, would be reflected in higher error rates for models on tasks dependent on causal links. The results of our study lend support to this hy- pothesis, suggesting that PLMs tend to priori- tize lexical patterns over higher-order signals. 1 Introduction Recognising causal links in narrative is an integral component of language comprehension that often relies on implicit cues (Trabasso and Sperry, 1985). Pre-trained language models, which form the basis of many language processing solutions nowadays, should therefore pick up on such cues and integrate them correctly with other signals to enable accurate causal inferences in downstream tasks, including question answering and information extraction. Psycholinguists have identified one such cue in the implicit causality bias of interpersonal verbs: some interpersonal verbs tend to implicate causal- ity on either their subject or their object (Garvey and Caramazza, 1974). It is this bias that leads to John�criticised�Mary � because���he�was�so�slow. John�apologised�to�Mary � because�he�was�wrong. S John O Mary EC he IC Subject- bias Object- bias Congruent�context Incongruent�context S John O Mary EC she IC EC IC John�criticised���Mary � because���h�was�fed�up. S John O Mary EC he IC Johapologised�to�Mary � because���he�felt�offended. O Mary S John she p(he | John praised Mary because) p(MASK = w | John praised Mary because MASK jiled the kippy w Figure 1: Illustration of implicit causality (IC) and ex- plicit causality (EC) in contexts where the two are con- gruent, and where they are incongruent. wide agreement among subjects in psycholinguis- tic studies, that in a sentence like John appreciates Mary, the cause for appreciation likely lies with a property or action of Mary’s rather than John’s. Causality can also be stated explicitly, in the form of a subordinate because clause, for instance, and it can optionally contradict the implicit causality of the verb in the main clause. In Figure …

Content truncated to 500 words