BERT Knows Punta Cana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7371–7385, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7371 BERT Knows Punta Cana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations Aina Gar´ı Soler Universit´e Paris-Saclay CNRS, LIMSI 91400 Orsay, France aina.gari@limsi.fr Marianna Apidianaki Department of Digital Humanities University of Helsinki Helsinki, Finland marianna.apidianaki@helsinki.fi Abstract Adjectives like pretty, beautiful and gorgeous describe positive properties of the nouns they modify but with different intensity. These dif- ferences are important for natural language understanding and reasoning. We propose a novel BERT-based approach to intensity de- tection for scalar adjectives. We model inten- sity by vectors directly derived from contextu- alised representations and show they can suc- cessfully rank scalar adjectives. We evaluate our models both intrinsically, on gold standard datasets, and on an Indirect Question Answer- ing task. Our results demonstrate that BERT encodes rich knowledge about the semantics of scalar adjectives, and is able to provide bet- ter quality intensity rankings than static em- beddings and previous models with access to dedicated resources. 1 Introduction Scalar adjectives describe a property of a noun at different degrees of intensity. Identifying the scalar relationship that exists between their meaning (for example, the increasing intensity between pretty, beautiful and gorgeous) is useful for text under- standing, for both humans and automatic systems. It can serve to define the sentiment and subjectiv- ity of a text, perform inference and textual entail- ment (Van Tiel et al., 2016; McNally, 2016), build question answering and recommendation systems (de Marneffe et al., 2010), and assist language learn- ers in distinguishing between semantically similar words (Sheinman and Tokunaga, 2009). We investigate the knowledge that the pre- trained BERT model (Devlin et al., 2019) encodes about the intensity expressed on an adjective scale. Given that this property is acquired by humans during language learning, we expect a language model (LM) exposed to massive amounts of text data during training to have also acquired some Figure 1: Full scale of adjectives describing positive and negative sentiment at different degrees from the SO-CAL dataset (Taboada et al., 2011). notion of adjective intensity. In what follows, we explore this hypothesis using representations ex- tracted from different layers of this deep neural model. Since the scalar relationship between adjec- tives is context-dependent (Kennedy and McNally, 2005) (e.g., what counts as tall may vary from context to context), we consider the contextualised representations produced by BERT to be a good fit for this task. We also propose a method inspired by gender bias work (Bolukbasi et al., 2016; Dev and Phillips, 2019) for detecting the intensity relation- ship of two adjectives on the fly. We view intensity as a direction in the semantic space which, once identified, can serve to determine the intensity of new adjectives. Our work falls in the neural network interpre- tation paradigm which explores the knowledge about language encoded in the representations of deep learning models (Voita et al., 2019a; Clark et al., …

Content truncated to 500 words