Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4741–4751 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4741 Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory Hannah Chen, Yangfeng Ji, David Evans Department of Computer Science University of Virginia Charlottesville, VA 22904 {yc4dx,yangfeng,evans}@virginia.edu Abstract Most NLP datasets are manually labeled, so suffer from inconsistent labeling or limited size. We propose methods for automatically improving datasets by viewing them as graphs with expected semantic properties. We con- struct a paraphrase graph from the provided sentence pair labels, and create an augmented dataset by directly inferring labels from the original sentence pairs using a transitivity property. We use structural balance theory to identify likely mislabelings in the graph, and flip their labels. We evaluate our methods on paraphrase models trained using these datasets starting from a pretrained BERT model, and find that the automatically-enhanced training sets result in more accurate models. 1 Introduction Having high quality annotated data is crucial for training supervised machine learning models. How- ever, producing large datasets with good labeling quality is expensive and labor intensive. Most NLP datasets rely on labels provided by human anno- tators with varying skills and limited training and expertise. The label instances are also often based on ambiguous definitions and guidelines. To address this problem, we study automated techniques to improve datasets for training and test- ing. In particular, we focus on paraphrase identifi- cation task, which aims to determine whether two given sentences are semantically equivalent. The sentences and labels in a dataset can be viewed as nodes and edges of a graph. Moving from single labeled sentence pairs to a graph provides a bet- ter understanding of the sentence relations of the dataset, which can be exploited to infer additional edge labels. In particular, since paraphrases are an equality relation, we can perform a transitive clo- sure on the graph to infer additional labels. In addi- tion, we use the notion of balance (Harary, 1953) for signed graphs to identify conflicted relations. In the context of semantic relationships between pairs of sentences, any paraphrases of a given sentence cannot be a non-paraphrase of each other since they should all share an identical meaning. Contributions. We show the benefits of represent- ing sentence pair relations as a graph. We first construct a paraphrase graph with the original pairs and their relation labels from the Quora Question Pairs (QQP) dataset (Iyer et al., 2017) following the structure of a signed graph. With the graph structure and the transitivity of paraphrases, we can automatically infer new sentence pair relations directly from the original dataset (Section 3.1). In addition, we identify and correct likely mislabeled pairs based on violations of expected structural balance properties we expect a valid paraphrase graph to satisfy (Section 3.2). We found 90 seem- ingly mislabeled sentence pairs in the QQP dataset. We show that fine-tuning a BERT model on the augmented set improves its performance on both the original and augmented …

Content truncated to 500 words