Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions

Proceedings of the 28th International Conference on Computational Linguistics, pages 5004–5012 Barcelona, Spain (Online), December 8-13, 2020 5004 Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions Matthijs Westera Universitat Pompeu Fabra / Leiden University matthijs.westera@gmail.com Jacopo Amidei The Open University jacopo.amidei@open.ac.uk Laia Mayol Universitat Pompeu Fabra laia.mayol@upf.edu Abstract We take a close look at a recent dataset of TED-talks annotated with the questions they implicitly evoke, TED-Q (Westera et al., 2020). We test to what extent the relation between a discourse and the questions it evokes is merely one of similarity or association, as opposed to deeper semantic/pragmatic interpretation. We do so by turning the TED-Q dataset into a binary classification task, constructing an analogous task from explicit questions we extract from the BookCorpus (Zhu et al., 2015), and fitting a BERT-based classifier alongside models based on different notions of similarity. The BERT-based classifier, achieving close to human perfor- mance, outperforms all similarity-based models, suggesting that there is more to identifying true evoked questions than plain similarity. 1 Introduction Asking explicit questions is essential for resolving information gaps, maintaining common ground, and fostering a shared view of the discourse topics. Understanding implicit questions is crucial for the same reasons, and moreover underlies our perception of discourse coherence (e.g., Van Kuppevelt (1995)): a discourse is coherent if each utterance addresses a question implicitly evoked by what came before. Despite their theoretical significance and their potential relevance to many Natural Language Processing (NLP) tasks, the questions implicitly evoked by a discourse have not received much attention in NLP, where the typical questions sought after are, rather, comprehension-testing questions – although this has begun to change, see Section 2. Recently a crowdsourced dataset became available aimed at making implicit questions explicit: the TED-Q dataset (Westera et al., 2020). We test to what extent the relation between a discourse and the questions it evokes is merely one if similarity or association, as opposed to deeper semantic/pragmatic interpretation. We do so by constructing a classification task from TED-Q where the goal is to predict, given a preceding discourse, whether the given question was evoked at that point or not. We compare human performance, a BERT-based classifier and four models based on different notions of similarity. In order to, moreover, investigate how the implicit questions of TED-Q compare to explicit questions from existing (unannotated) corpora, which are far easier to come by, we construct an analogous task from written dialogue in the existing BookCorpus (Zhu et al., 2015).1 2 Background Questions have long been an important topic in NLP, but mostly in their role of comprehension-testing questions, i.e., questions that are answered by the discourse, rather than unanswered questions that it evokes, as in the current work. Recently this has begun to change. For example, in terms of available datasets, the QuAC (’Question Answering in Context’) dataset (Choi et al., 2018) consists of 100K ques- tions asked in unscripted dialogue about a given topic; Rao and Daum´e III (2018) present a dataset of …

Content truncated to 500 words