Learning Disentangled Representations of Negation and Uncertainty

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8380 - 8397 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Disentangled Representations of Negation and Uncertainty Jake Vasilakes1, Chrysoula Zerva2,3, Makoto Miwa4,5, Sophia Ananiadou1,5 1National Centre for Text Mining, Department of Computer Science, The University of Manchester 2Instituto Superior Técnico, 3Instituto de Telecomunicações 4Toyota Technological Institute 5Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology {jake.vasilakes, sophia.ananiadou}@manchester.ac.uk, chrysoula.zerva@tecnico.ulisboa.pt makoto-miwa@toyota-ti.ac.jp Abstract Negation and uncertainty modeling are long- standing tasks in natural language processing. Linguistic theory postulates that expressions of negation and uncertainty are semantically inde- pendent from each other and the content they modify. However, previous works on represen- tation learning do not explicitly model this in- dependence. We therefore attempt to disentan- gle the representations of negation, uncertainty, and content using a Variational Autoencoder1. We find that simply supervising the latent rep- resentations results in good disentanglement, but auxiliary objectives based on adversarial learning and mutual information minimization can provide additional disentanglement gains. 1 Introduction In formal semantics, negation and uncertainty are operators whose semantic functions are inde- pendent of the propositional content they modify (Cann, 1993a,b)2. That is, it is possible to form flu- ent statements by varying only one of these aspects while leaving the others the same. Negation, un- certainty, and content can thus be viewed as disen- tangled generative factors of knowledge and belief statements (see Figure 1). Disentangled representation learning (DRL) of factors of variation can improve the robustness of representations and their applicability across tasks (Bengio et al., 2013). Specifically, negation and un- certainty are important for downstream NLP tasks such as sentiment analysis (Benamara et al., 2012; Wiegand et al., 2010), question answering (Yatskar, 2019; Yang et al., 2016), and information extrac- tion (Stenetorp et al., 2012). Disentangling nega- 1We make our implementation and data avail- able at https://github.com/jvasilakes/ disentanglement-vae 2Specifically, the propositional content can be represented by a variable, such as ¬p. Trees might not have leaves. Figure 1: Example indicating the distinction between uncertainty, negation and content. The content “Trees have leaves” is modified by the negation (bold) and uncertainty (underlined) factors. tion and uncertainty can therefore provide robust representations for these tasks, and disentangling them from content can assist tasks that rely on core content preservation such as controlled generation (Logeswaran et al., 2018) and abstractive summa- rization (Maynez et al., 2020). Still, no previous work has tested whether nega- tion, uncertainty, and content can be disentangled, as linguistic theory suggests, although previous works have disentangled attributes such as syntax, semantics, and style (Balasubramanian et al., 2021; John et al., 2019; Cheng et al., 2020b; Bao et al., 2019; Hu et al., 2017; Colombo et al., 2021). To fill this gap, we aim to answer the following research questions: RQ1: Is it possible to estimate a model of statements that upholds the proposed statistical independence between negation, uncertainty, and content? RQ2: A number of existing disentanglement objectives have …

Content truncated to 500 words