Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2792–2798 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2792 Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets Nicole Peinelt1,2 and Maria Liakata1,2 and Dong Nguyen1,3 1The Alan Turing Institute, London, UK 2University of Warwick, Coventry, UK 3Utrecht University, Utrecht, The Netherlands {n.peinelt, m.liakata}@warwick.ac.uk, dnguyen@turing.ac.uk Abstract Existing datasets for scoring text pairs in terms of semantic similarity contain instances whose resolution differs according to the degree of difficulty. This paper proposes to distinguish obvious from non-obvious text pairs based on superficial lexical overlap and ground-truth labels. We characterise existing datasets in terms of containing difficult cases and find that recently proposed models struggle to capture the non-obvious cases of semantic similarity. We describe metrics that emphasise cases of similarity which require more complex infer- ence and propose that these are used for eval- uating systems for semantic similarity. 1 Introduction Modelling semantic similarity between a pair of texts is a fundamental task in NLP with a wide range of applications (Baudiˇs et al., 2016). One area of active research is Community Question Answering (CQA) (Nakov et al., 2017; Bonadi- man et al., 2017), which is concerned with the au- tomatic answering of questions based on user gen- erated content from Q&A websites (e.g. StackEx- change) and requires modelling the semantic simi- larity between question and answer pairs. Another well-studied task is paraphrase detection (Socher et al., 2011; He et al., 2015; Tomar et al., 2017), which models the semantic equivalence between a pair of sentences. Evaluation for such tasks has primarily fo- cused on metrics, such as mean average preci- sion (MAP), F1 or accuracy, which give equal weights to all examples, regardless of their diffi- culty. However, as illustrated by the examples in Table 1, not all items within text pair similarity datasets are equally difficult to resolve. Recent work has shown the need to better un- derstand limitations of current models and datasets in natural language understanding (Wadhwa et al., id case documents 160174 Po what‘s the origin of the word o‘clock? what is the origin of the word o‘clock? 115695 Pn which is the best way to learn coding? how do you learn to program? 193190 No what are the range of careers in biotechnology in indonesia? how do you tenderize beef stew meat? 268368 Nn what is meant by ‘e‘ in mathematics? what is meant by mathematics? Table 1: Examples for difficulty cases from the devel- opment set of the Quora dataset. o=obvious, n=non- obvious, N=negative label, P=positive label 2018a; Rajpurkar et al., 2018). For example, Kaushik and Lipton (2018) showed that models sometimes exploit dataset properties to achieve high performance even when crucial task infor- mation is withheld, and Gururangan et al. (2018) demonstrated that model performance is inflated by annotation artefacts in natural language infer- ence tasks. In this paper, we analyse current datasets and re- cently proposed models by focusing on item …

Content truncated to 500 words