Assessing the Eligibility of Backtranslated Samples Based on Semantic Similarity for the Paraphrase Identification Task

Proceedings of Recent Advances in Natural Language Processing, pages 301–308 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_035 301 Assessing the Eligibility of Backtranslated Samples Based on Semantic Similarity for the Paraphrase Identification Task Jean-Philippe Corbeil and Hadi Abdi Ghavidel Polytechnique Montreal {jean-philippe.corbeil,hadi.abdi-ghavidel}@polymtl.ca Abstract In the domain of natural language augmenta- tion, the eligibility of generated samples re- mains not well understood. To gather in- sights around this eligibility issue, we ap- ply a transformer-based similarity calculation within the BET framework based on backtrans- lation, in the context of automated paraphrase detection. While providing a rigorous statisti- cal foundation to BET, we push their results by analyzing statistically the impacts of the level of qualification, and several sample sizes. We conducted a vast amount of experiments on the MRPC corpus using six pre-trained mod- els: BERT, XLNet, Albert, RoBERTa, Electra, and DeBerta. We show that our method im- proves significantly these ”base” models while using only a fraction of the corpus. Our results suggest that using some of those smaller pre- trained models, namely RoBERTa base and Electra base, helps us reach F1 scores very close to their large counterparts, as reported on the GLUE benchmark. On top of acting as a regularizer, the proposed method is efficient in dealing with data scarcity with improvements of around 3% in F1 score for most pre-trained models, and more than 7.5% in the case of Electra. 1 Introduction Natural language processing (NLP) tasks require sufficiently large datasets to achieve the maximum robustness of the trained models. Low sizes of data pose the risks of hindering the models’ con- vergence during the training process, which leads to less accurate predictions (e.g. classification) or generations (e.g. translations). On the other hand, the provision of high-quality labelled data is often very expensive both in terms of money and time. As a result, NLP scientists seek alternative methods to tackle this issue. One of the solutions is the ap- plication of data augmentation techniques. These methods help considerably to alleviate insufficien- cies regarding the quantity of labelled data and the expertise to annotate the data. These augmentation techniques are also proved to induce a regulariza- tion effect during the training of NLP models to avoid overfitting, most notably, on surface cues. In this paper, we examine the impact of adding a post-processing stage after applying such data aug- mentation technique to assess the eligibility of the generated samples. We intend to run our analysis in automated paraphrase identification. In this regard, we increase the size of the paraphrase data through a backtranslation method called BET (Corbeil and Abdi Ghavidel, 2020). In particular, we conduct the following experiments: • We take randomly several samples of the orig- inal train set and augment them with back- translation. • After augmenting the textual data, we assess the eligibility by applying a similarity filter. We report the results for three criteria: 0.8, 0.9, and 0.95. • We examine six pre-trained transformer mod- els: BERT, XLNet, RoBERTa, ALBERT, Electra, and DeBerta. • We run ten …

Content truncated to 500 words