Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208–4213 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4208 Do you know that Florence is packed with visitors? Evaluating state-of-the-art models of speaker commitment Nanjiang Jiang and Marie-Catherine de Marneffe Department of Linguistics The Ohio State University {jiang.1879, demarneffe.1}@osu.edu Abstract When a speaker, Mary, asks Do you know that Florence is packed with visitors?, we take her to believe that Florence is packed with visitors, but not if she asks Do you think that Florence is packed with visitors? Inferring speaker commitment (aka event factuality) is crucial for information extraction and question an- swering. Here we explore the hypothesis that linguistic deficits drive the error patterns of speaker commitment models by analyzing the linguistic correlates of model errors on a chal- lenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The Com- mitmentBank is annotated with speaker com- mitment towards the content of the comple- ment (Florence is packed with visitors in our example) of clause-embedding verbs (know, think) under four entailment-canceling envi- ronments. We found that a linguistically- informed model outperforms a LSTM-based one, suggesting that linguistic knowledge is needed to capture such challenging naturalis- tic data. A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic construc- tions (e.g., conditionals) in natural language, highlighting directions for improvement. 1 Introduction Prediction of speaker commitment1 is the task of determining to what extent the speaker is com- mitted to an event in a sentence as actual, non- actual, or uncertain. This matters for downstream NLP applications, such as information extraction 1Previous work uses event factuality, verdicality, or com- mitted belief; the terms refer to the same linguistic phe- nomenon, perhaps with different emphasis. or question answering: for instance, we should ex- tract from example (1) in Table 1 that the speaker could wish someone dead, but from (3) that people should not be allowed to carry guns in their vehi- cles, even though both events are embedded under believe and negation. There has been work on factors leading to speaker commitment in theoretical linguistics (i.a., Karttunen (1971); Simons et al. (2010)) and com- putational linguistics (i.a., Diab et al. (2009); Saur´ı and Pustejovsky (2012); Prabhakaran et al. (2015)), but mostly on constructed or newswire examples, which may simplify the task by fail- ing to reflect the lexical and syntactic diversity of naturally occurring utterances. de Marneffe et al. (2019) introduced the CommitmentBank, a dataset of naturally occurring sentences anno- tated with speaker commitment towards the con- tent of complements of clause-embedding verbs under canceling-entailment environments (nega- tion, modal, question and conditional), to study the linguistic correlates of speaker commitment. In this paper, we use it to evaluate two state-of-the-art (SoA) models of speaker commitment: Stanovsky et al. (2017) and Rudinger …

Content truncated to 500 words