Attention Is (not) All You Need for Commonsense Reasoning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4831–4836 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4831 Attention Is (not) All You Need for Commonsense Reasoning Tassilo Klein1, Moin Nabi1 1SAP Machine Learning Research, Berlin, Germany {tassilo.klein, m.nabi}@sap.com Abstract The recently introduced BERT model exhibits strong performance on several language un- derstanding benchmarks. In this paper, we describe a simple re-implementation of BERT for commonsense reasoning. We show that the attentions produced by BERT can be directly utilized for tasks such as the Pronoun Dis- ambiguation Problem and Winograd Schema Challenge. Our proposed attention-guided commonsense reasoning method is conceptu- ally simple yet empirically powerful. Exper- imental analysis on multiple datasets demon- strates that our proposed system performs re- markably well on all cases while outperform- ing the previously reported state of the art by a margin. While results suggest that BERT seems to implicitly learn to establish complex relationships between entities, solving com- monsense reasoning tasks might require more than unsupervised models learned from huge text corpora. 1 Introduction Recently, neural models pre-trained on a lan- guage modeling task, such as ELMo (Peters et al., 2018b), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impres- sive results on various natural language processing tasks such as question-answering and natural lan- guage inference. The success of BERT can largely be associated to the notion of context-aware word embeddings, which differentiate it from common approaches such as word2vec (Mikolov et al., 2013) that establish a static semantic embedding. Since the introduction of BERT, the NLP com- munity continues to be impressed by the amount of ideas produced on top of this powerful lan- guage representation model. However, despite its success, it remains unclear whether the repre- sentations produced by BERT can be utilized for tasks such as commonsense reasoning. Particu- larly, it is not clear whether BERT shed light on solving tasks such as the Pronoun Disambigua- tion Problem (PDP) and Winograd Schema Chal- lenge (WSC). These tasks have been proposed as potential alternatives to the Turing Test, because they are formulated to be robust to statistics of word co-occurrence (Levesque et al., 2012). Below is a popular example from the binary- choice pronoun coreference problem (Lee et al., 2017) of WSC: Sentence: The trophy doesn't fit in the suit- case because it is too small. Answers: A) the trophy B) the suitcase Humans resolve the pronoun “it” to “the suit- case” with no difficulty, whereas a system without commonsense reasoning would be unable to dis- tinguish “the suitcase” from the otherwise viable candidate, “the trophy”. Previous attempts at solving WSC usually in- volve heavy utilization of annotated knowledge bases (KB), rule-based reasoning, or hand-crafted features (Peng et al., 2015; Bailey et al., 2015; Sch¨uller, 2014; Sharma et al., 2015; Morgenstern et al., 2016). There are also some empirical works towards solving WSC making use of learn- ing (Rahman and Ng, 2012; Tang et al., 2018; …

Content truncated to 500 words