Empowering Language Understanding with Counterfactual Reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2226–2236 August 1–6, 2021. ©2021 Association for Computational Linguistics 2226 Empowering Language Understanding with Counterfactual Reasoning Fuli Feng1,2, Jizhi Zhang3, Xiangnan He3∗, Hanwang Zhang4, Tat-Seng Chua2 1NExT-Sea Joint Lab, 2National University of Singapore 3University of Science and Technology of China, 4Nanyang Technological University fulifeng93@gmail.com, cdzhangjizhi@mail.ustc.edu.cn xiangnanhe@gmail.com,hanwangzhang@ntu.edu.sg,dcscts@nus.edu.sg Abstract Present language understanding methods have demonstrated extraordinary ability of recog- nizing patterns in texts via machine learning. However, existing methods indiscriminately use the recognized patterns in the testing phase that is inherently different from us humans who have counterfactual thinking, e.g., to scru- tinize for the hard testing samples. Inspired by this, we propose a Counterfactual Reason- ing Model, which mimics the counterfactual thinking by learning from few counterfactual samples. In particular, we devise a generation module to generate representative counterfac- tual samples for each factual sample, and a ret- rospective module to retrospect the model pre- diction by comparing the counterfactual and factual samples. Extensive experiments on sentiment analysis (SA) and natural language inference (NLI) validate the effectiveness of our method. 1 Introduction Language understanding (Ke et al., 2020) is a central theme of artificial intelligence (Chomsky, 2002), which empowers a wide spectral of applica- tions such as sentiment evaluation (Feldman, 2013), commonsense inference (Bowman et al., 2015). The models are trained on labeled data to recognize the textual patterns closely correlated to different labels. Owing to the extraordinary representational capacity of deep neural networks, the models can well recognize the pattern and make prediction ac- cordingly (Devlin et al., 2019). However, the cog- nitive ability of these data-driven models is still far from human beings due to lacking counterfactual thinking (Pearl, 2019). Counterfactual thinking is a high-level cognitive ability beyond pattern recognition (Pearl, 2019). In addition to observing the patterns within factual ∗Corresponding author. samples, counterfactual thinking calls for compar- ing the fact with imaginations, so as to make bet- ter decision. For instance, given a factual sample “What do lawyers do when they die? Lie still.”, the intuitive evaluation of its sentiment based on the textual patterns will recognize “Lie still” as an objective description of body posture which is neutral. By scrutinizing that the “still” could be intentionally postposed, we can imagine a counter- factual sample “What do lawyers do when they die? Still lie.” and uncover the negative sarcastic pun, whose sentiment is more accurate. Recent work (Kaushik et al., 2019; Zeng et al., 2020) shows that incorporating counterfactual sam- ples into model training improves the generaliza- tion ability. However, these methods follow the standard machine learning paradigm that uses the same procedure (e.g., a forward propagation) to make prediction in the testing phase. That is, mak- ing decision for testing samples according to their relative positions to the model decision boundary. The indiscriminate procedure focuses on the textual patterns occurred in the testing sample and treats all testing samples equally, which easily fails on hard samples (cf. Figure 1). On the contrary, humans can discriminate hard samples …

Content truncated to 500 words