A Gold Standard for Scalar Adjectives

A Gold Standard for Scalar Adjectives Bryan Wilkinson and Tim Oates University of Maryland, Baltimore County 1000 Hilltop Circle, Baltimore, MD, 21250 bryan.wilkinson@umbc.edu, oates@cs.umbc.edu Abstract We present a gold standard for evaluating scale membership and the order of scalar adjectives. In addition to evaluating existing methods of ordering adjectives, this knowledge will aid in studying the organization of adjectives in the lexicon. This resource is the result of two elicitation tasks conducted with informants from Amazon Mechanical Turk. The first task is notable for gathering open-ended lexical data from informants. The data is analyzed using Cultural Consensus Theory, a framework from anthropology, to not only determine scale membership but also the level of consensus among the informants (Romney et al., 1986). The second task gathers a culturally salient ordering of the words determined to be members. We use this method to produce 12 scales of adjectives for use in evaluation. Keywords: scalar adjective, cultural consensus theory (CCT), crowdsourcing 1. Introduction Scalar adjectives like warm, big, and good represent a value on the scales of TEMPERATURE, SIZE, and QUALITY re- spectively. Kennedy and McNally have shown that the semantics of individual words can be mapped to degrees (2005). The language modeling community has questioned whether this knowledge should be represented in lexicons, such as WordNet, and how it should be learned. Figure 1 shows one potential representation of words on a scale for SIZE, with each word being placed on a continuum of val- ues for the property. To that end, several proposals have been made on how to learn the scalar relationship between two or more words. Sheinman, et al. propose the use of lexico-syntactic pat- terns to determine the ordering of the words contained in one WordNet adjective grouping (2013). De Melo and Bansal extended this work by summing over occurrences of the patterns containing pairs of words as a scoring func- tion. They then apply Mixed Integer Linear Programming to determine the global ordering among a group of words (De Melo and Bansal, 2013). Kim and de Marneffe take a word embedding approach to the problem, finding words closest to the mean and quartile points along the line be- tween two embeddings (2013). Figure 1: Example of a scale for SIZE. In this paper we present a gold standard of 12 adjective scales for use in evaluation of these methods as well as for use in investigating scalar implicature, a need highlighted by Van Tiel et al (2016) 1. We use cultural consensus the- ory (CCT) to both produce the gold standard as well as to gain insight on the level of consensus among the informants (Romney et al., 1986). 1Available at https://github.com/Coral-Lab/scales CCT was developed to aggregate the shared knowledge of a domain by a culture (Weller, 2007). It has roots in test theory and was developed as an analysis of latent variables of participants that can be done when true answers are un- known as opposed to other methods such as Classical Test Theory or Item …

Content truncated to 500 words