Approximating Givenness in Content Assessment through Distributional Semantics

Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM 2016), pages 209–218, Berlin, Germany, August 11-12, 2016. Approximating Givenness in Content Assessment through Distributional Semantics Ramon Ziai Kordula De Kuthy Detmar Meurers Collaborative Research Center 833 University of T¨ubingen {rziai,kdk,dm}@sfs.uni-tuebingen.de Abstract Givenness (Schwarzschild, 1999) is one of the central notions in the formal pragmatic literature discussing the organization of dis- course. In this paper, we explore where distributional semantics can help address the gap between the linguistic insights into the formal pragmatic notion of Givenness and its implementation in computational linguistics. As experimental testbed, we focus on short answer assessment, in which the goal is to assess whether a student response correctly answers the provided reading comprehen- sion question or not. Current approaches only implement a very basic, surface-based perspective on Givenness: A word of the answer that appears as such in the question counts as GIVEN. We show that an approach approximating Givenness using distributional semantics to check whether a word in a sentence is similar enough to a word in the context to count as GIVEN is more successful quanti- tatively and supports interesting qualitative insights into the data and the limitations of a basic distributional semantic approach identifying Givenness at the lexical level. 1 Introduction Givenness is one of the central notions in the formal pragmatic literature discussing the organization of discourse. The distinction between given and new material in an utterance dates back at least to Hal- liday (1967) where given is defined as “anaphori- cally recoverable” and the notion is used to predict patterns of prosodic prominence. Schwarzschild (1999) proposes to define Givenness in terms of the entailment of the existential f-closure between previously mentioned material and the GIVEN ex- pression, hereby also capturing the occurrence of synonyms and hyponyms as given. On the theoretical linguistic side, a foundational question is whether an approach to Information Structure should be grounded in terms of a Given- New or a Focus-Background dichotomy, or whether the two are best seen as complementing each other. Computational linguistic research on short answer assessment points in the direction of both perspec- tives providing performance gains (Ziai and Meur- ers, 2014). On the empirical side, the characteristic problem of obtaining high inter-annotator agree- ment in focus annotation (Ritz et al., 2008; Cal- houn et al., 2010) can be overcome through an incremental annotation process making reference to questions as part of an explicit task context (Ziai and Meurers, 2014; De Kuthy et al., 2016). In short answer assessment approaches determin- ing whether a student response correctly answers a provided reading comprehension question, the practical value of excluding material that is men- tioned in the question from evaluating the content of the answer has been clearly established (Meurers et al., 2011; Mohler et al., 2011). Yet these com- putational linguistic approaches only implement a very basic, completely surface-based perspective on Givenness: A word of the answer that appears as such in the question counts as GIVEN. Such a surface-based approach …

Content truncated to 500 words