Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2865–2876, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2865 Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts Sandro Pezzelle and Raquel Fern´andez Institute for Logic, Language, and Computation University of Amsterdam {s.pezzelle|raquel.fernandez}@uva.nl Abstract This work aims at modeling how the meaning of gradable adjectives of size (‘big’, ‘small’) can be learned from visually-grounded con- texts. Inspired by cognitive and linguistic ev- idence showing that the use of these expres- sions relies on setting a threshold that is de- pendent on a specific context, we investigate the ability of multi-modal models in assess- ing whether an object is ‘big’ or ‘small’ in a given visual scene. In contrast with the standard computational approach that simplis- tically treats gradable adjectives as ‘fixed’ at- tributes, we pose the problem as relational: to be successful, a model has to consider the full visual context. By means of four main tasks, we show that state-of-the-art models (but not a relatively strong baseline) can learn the func- tion subtending the meaning of size adjectives, though their performance is found to decrease while moving from simple to more complex tasks. Crucially, models fail in developing abstract representations of gradable adjectives that can be used compositionally. 1 Introduction There is no doubt that planets are big things. Among the planets of our Solar System, however, Mars is unquestionably a small planet (though not the smallest), while Saturn is definitely a big one (though not the biggest). This example high- lights some crucial properties of gradable adjec- tives (hence, GAs). First, what counts as ‘big’ or ‘small’ is relative, i.e., determined by the context: Phobos is both a big moon of Mars and a small celestial body of the Solar System. This makes GAs different from non-gradable or absolute ad- jectives like ‘open’, ‘empty’, ‘red’: Mars, for ex- ample, is red and rocky in any circumstance. More formally, the compositional semantic properties of GAs are subsective since they select a subset of en- tities denoted by the noun they modify, which acts as a reference set (∥big∥ ⊆ ∥moon∥), while non- GAs are intersective (∥Galilean∥ ∩ ∥moon∥). This has consequences for the inferences they license: if Ganymede is both a Galilean moon and a celes- tial body, we can infer that Ganymede is a Galilean celestial body. In contrast, if it is a big moon and a celestial body, the inference that Ganymede is a big celestial body is not valid (Partee, 1995). Second, besides depending on a contextually- given reference set, GAs rely on orderings, i.e, they denote functions that map entities onto scales of degrees (Cresswell, 1976; Kennedy, 1999). Us- ing ‘big’ or ‘small’, thus, implies mapping a target object onto a size scale, which allows us to use degree morphology to express that Saturn is big- ger than Mars (comparative form) or that Mercury is the …

Content truncated to 500 words