Enhanced Metaphor Detection via Incorporation of External Knowledge Based on Linguistic Theories

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 103–119 August 1–6, 2021. ©2021 Association for Computational Linguistics 103 SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics Hitomi Yanaka1,2, Koji Mineshima3, and Kentaro Inui4,2 1The University of Tokyo, 2RIKEN, 3Keio University, 4Tohoku University hyanaka@is.s.u-tokyo.ac.jp, minesima@abelard.flet.keio.ac.jp, inui@ecei.tohoku.ac.jp Abstract Recently, deep neural networks (DNNs) have achieved great success in semantically chal- lenging NLP tasks, yet it remains unclear whether DNN models can capture composi- tional meanings, those aspects of meaning that have been long studied in formal semantics. To investigate this issue, we propose a Sys- tematic Generalization testbed based on Natu- ral language Semantics (SyGNS), whose chal- lenge is to map natural language sentences to multiple forms of scoped meaning representa- tions, designed to account for various semantic phenomena. Using SyGNS, we test whether neural networks can systematically parse sen- tences involving novel combinations of logi- cal expressions such as quantifiers and nega- tion. Experiments show that Transformer and GRU models can generalize to unseen combi- nations of quantifiers, negations, and modifiers that are similar to given training instances in form, but not to the others. We also find that the generalization performance to unseen com- binations is better when the form of meaning representations is simpler. The data and code for SyGNS are publicly available at https: //github.com/verypluming/SyGNS. 1 Introduction Deep neural networks (DNNs) have shown im- pressive performance in various language under- standing tasks (Wang et al., 2019a,b, i.a.), in- cluding semantically challenging tasks such as Natural Language Inference (NLI; Dagan et al., 2013; Bowman et al., 2015). However, a number of studies to probe DNN models with various NLI datasets (Naik et al., 2018; Dasgupta et al., 2018; Yanaka et al., 2019; Kim et al., 2019; Richardson et al., 2020; Saha et al., 2020; Geiger et al., 2020) have reported that current DNN models have some limitations to generalize to diverse semantic phe- nomena, and it is still not clear whether DNN mod- Training Sentences One wild dog ran All dogs ran One dog did not run Generalization Test All wild dogs ran All dogs did not run MODIFIER QUANTIFIER NEGATION Multiple meaning representations MR1: ∀x.(dog↓(x) ∧ wild↓(x)) → (run↑(x)) MR2: ALL AND DOG WILD RUN MR3: x1 wild(x1) dog(x1) ⇒ run(x1) Evaluation methods Exact matching: G = P? Theorem Proving: G ⇔ P? Polarity: {dog↓, wild↓, run↑} Clausal form: b1 IMP b2 b3 b2 REF x1 b2 wild x1 b2 dog x1 b3 run x1 Figure 1: Illustration of our evaluation protocol using SyGNS. The goal is to map English sentences to mean- ing representations. The generalization test evaluates novel combinations of operations (modifier, quantifier, negation) in the training set. We use multiple meaning representations and evaluation methods. els obtain the ability to capture compositional as- pects of meaning in natural language. There are two issues to consider here. First, re- cent analyses (Talmor and Berant, 2019; Liu et al., 2019; McCoy et al., 2019) have pointed out that the standard paradigm for evaluation, …

Content truncated to 500 words