Contrast Sets for Stativity of English Verbs in Context

Proceedings of the 29th International Conference on Computational Linguistics, pages 4028–4036 October 12–17, 2022. 4028 Contrast Sets for Stativity of English Verbs in Context Daniel Chen & Alexis Palmer Department of Linguistics University of Colorado {daniel.chen-1, alexis.palmer}@colorado.edu Abstract For the task of classifying verbs in context as dynamic or stative, current models approach human performance, but only for particular data sets. To better understand the performance of such models, and how well they are able to generalize beyond particular test sets, we apply the contrast set (Gardner et al., 2020) methodology to stativity classification. We cre- ate nearly 300 contrastive pairs by perturbing test set instances just enough to change their labels from one class to the other, while preserv- ing coherence, meaning, and well-formedness. Contrastive evaluation shows that a model with near-human performance on an in-distribution test set degrades substantially when applied to transformed examples, showing that the stative vs. dynamic classification task is more complex than the model performance might otherwise suggest. Code and data are freely available.1 1 Introducing stativity Aspectual properties of verbs, and the clauses they inhabit, have the potential to support a range of natural language processing tasks, such as event ordering (Modi and Titov, 2014) and temporal re- lation classification (Costa and Branco, 2012), as well as contributing to speaker choices around situ- ational construal (Trott et al., 2020). At the same time, verb and situational aspect are a complex set of interacting properties, in which the meaning of the verb, the nature of its arguments, adverbial modifiers, and grammatical features such as verb tense and nominal definiteness can all play a role in determining the aspectual make-up of a clause. This sensitivity of aspectual categorization to small shifts in linguistic form is one reason that automatic prediction of aspectual classes is an espe- cially challenging computational problem. In this paper we explore the stability of automatic classifi- cation for one particular facet of aspect: stativity of 1https://github.com/dchensta/se_ contrast English verbs. Stativity reflects the degree to which a verb represents a static situation versus a situation that reflects some degree of dynamicity. Dynamic verbs typically involve some change of state. (1-3) below show examples of the three classes relevant for our study: DYNAMIC verbs, STATIVE verbs, and verbs for which annotators CANNOT_DECIDE. (1) Table 7 shows results from the latest experi- ments. STATIVE (2) Dr. Smith showed her students how to work with the new GPUs. DYNAMIC (3) The earlier paper shows the effectiveness of in- corporating linguistic features. CANNOT_DECIDE In (1), the table is static, and the results exist in the table; no change of state is indicated. (2) highlights the dynamic sense of show, in which the Agent is giving a demonstration. (3) allows two readings. In the STATIVE reading, the result about linguistic features is a static property of the paper; it simply exists in the paper. In the DYNAMIC read- ing, the paper demonstrates the effectiveness of linguistic features through an argument that devel- ops and progresses over the …

Content truncated to 500 words