PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7572–7582, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7572 PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge Yun He, Zhuoer Wang, Yin Zhang, Ruihong Huang, James Caverlee Department of Computer Science and Engineering, Texas A&M University {yunhe, wang, zhan13679, huangrh, caverlee}@tamu.edu Abstract We present a new benchmark dataset called PARADE for paraphrase identification that re- quires specialized domain knowledge. PA- RADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non- paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annota- tors have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that in- corporate domain knowledge. We make our data and code freely available.1 1 Introduction Paraphrases are sentences that express the same (or similar) meaning by using different wording (Bha- gat and Hovy, 2013). Automatically identifying paraphrases and non-paraphrases has proven useful for a wide range of natural language processing (NLP) applications, including question answering, semantic parsing, information extraction, machine translation, textual entailment, and semantic textual similarity. Paraphrase identification (PI) is typically formal- ized as a binary classification problem: given two sentences, determine if they roughly express the same meaning. Traditional paraphrase identifica- tion approaches (Mihalcea et al., 2006; Kozareva and Montoyo, 2006; Wan et al., 2006; Das and Smith, 2009; Xu et al., 2014) mainly rely on lex- ical and syntactic overlap features to measure the 1https://github.com/heyunh2015/PARADE_ dataset semantic similarity between the two sentences. Ex- amples include string-based features (e.g., whether two sentences share the same words), part-of- speech features (e.g., whether shared words have the same POS tags), and dependency-based fea- tures (e.g., whether two sentences have similar de- pendency trees). s1: the lowest level of code made up of 0s and 1s. s2: binary instructions used by the cpu. Label: paraphrase (both describe “Machine Code”) s3: a graph representation that uses a 2d array such that if arr[i][j] == 1, there is an edge between vertices i and j s4: a matrix which records the number of direct links between vertices Label: paraphrase (both describe “Adjacency Matrix”) s5: how the optimal solution to a linear programming problem changes as the problem data are modified. s6: how changes in the coefficients of a linear programming problem affect the optimal solution Label: non-paraphrase Table 1: Examples of paraphrases and non-paraphrases from the computer science domain. Judgments are made based on domain knowledge rather than lexical or syntactic features. Overlapping words (other than stop-words) are in bold and key different words are un- derlined. However, these shallow lexical and syntactic overlap features may …

Content truncated to 500 words