Unsupervised Deep Structured Semantic Models for Commonsense Reasoning

Proceedings of NAACL-HLT 2019, pages 882–891 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 882 Unsupervised Deep Structured Semantic Models for Commonsense Reasoning Shuohang Wang1∗, Sheng Zhang2, Yelong Shen4, Xiaodong Liu3, Jingjing Liu3, Jianfeng Gao3, Jing Jiang1 1Singapore Management University,2Johns Hopkins University, 3Microsoft, 4Tencent AI Lab {shwang.2014,jingjiang}@smu.edu.sg, zsheng2@jhu.edu {xiaodl,jingjl,jfgao}@microsoft.com, yelongshen@tencent.com Abstract Commonsense reasoning is fundamental to natural language understanding. While tra- ditional methods rely heavily on human- crafted features and knowledge bases, we ex- plore learning commonsense knowledge from a large amount of raw text via unsupervised learning. We propose two neural network models based on the Deep Structured Seman- tic Models (DSSM) framework to tackle two classic commonsense reasoning tasks, Wino- grad Schema challenges (WSC) and Pronoun Disambiguation (PDP). Evaluation shows that the proposed models effectively capture con- textual information in the sentence and co- reference information between pronouns and nouns, and achieve significant improvement over previous state-of-the-art approaches. 1 Introduction Commonsense reasoning is concerned with sim- ulating the human ability to make presumptions about the type and essence of ordinary situa- tions they encounter every day (Davis and Mar- cus, 2015). It is one of the key challenges in nat- ural language understanding, and has drawn in- creasing attention in recent years (Levesque et al., 2011; Roemmele et al., 2011; Zhang et al., 2017; Rashkin et al., 2018a,b; Zellers et al., 2018; Trinh and Le, 2018). However, due to the lack of la- beled training data or comprehensive hand-crafted knowledge bases, commonsense reasoning tasks such as Winograd Schema Challenge (Levesque et al., 2011) are still far from being solved. In this work, we propose two effective unsu- pervised models for commonsense reasoning, and evaluate them on two classic commonsense rea- soning tasks: Winograd Schema Challenge (WSC) and Pronoun Disambiguation Problems (PDP). Compared to other commonsense reasoning tasks, ∗Work done when the author was at Microsoft 1. The city councilmen refused the demonstrators a permit because they feared violence. Who feared violence? A. The city councilmen B. The demonstra- tors 2. The city councilmen refused the demonstrators a permit because they advocated violence. Who advocated violence? A. The city councilmen B. The demonstra- tors Table 1: Examples from Winograd Schema Challenge (WSC). The task is to identify the reference of the pro- noun in bold. WSC and PDP better approximate real human rea- soning, and can be more easily solved by native English-speaking adults (Levesque et al., 2011). In addition, they are also technically challenging. For example, the best reported result on WSC is only 20 percentage points better than random guess in accuracy (Radford et al., 2019). Table 1 shows two examples from WSC. In or- der to resolve the co-reference in these two exam- ples, one cannot predict what “they” refers to un- less she is equipped with the commonsense knowl- edge that “demonstrators usually cause violence and city councilmen usually fear violence”. As no labeled training data is available for these tasks, previous approaches are based on either hand-crafted knowledge bases …

Content truncated to 500 words