Pre-trained Language Model Based Active Learning for Sentence Matching

Proceedings of the 28th International Conference on Computational Linguistics, pages 1495–1504 Barcelona, Spain (Online), December 8-13, 2020 1495 Pre-trained Language Model Based Active Learning for Sentence Matching Guirong Bai1,2, Shizhu He1,2, Kang Liu1,2, Jun Zhao1,2, Zaiqing Nie3 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences 2 School of Artificial Intelligence, University of Chinese Academy of Sciences 3 Alibaba AI Labs {guirong.bai, shizhu.he, kliu, jzhao}@nlpr.ia.ac.cn zaiqing.nzq@alibaba-inc.com Abstract Active learning is able to significantly reduce the annotation cost for data-driven techniques. However, previous active learning approaches for natural language processing mainly depend on the entropy-based uncertainty criterion, and ignore the characteristics of natural language. In this paper, we propose a pre-trained language model based active learning approach for sentence matching. Differing from previous active learning, it can provide linguistic criteria from the pre-trained language model to measure instances and help select more effective instances for annotation. Experiments demonstrate our approach can achieve greater accuracy with fewer labeled training instances. 1 Introduction Sentence matching is a fundamental technology in natural language processing. Over the past few years, deep learning as a data-driven technique has yielded state-of-the-art results on sentence matching (Wang et al., 2017; Chen et al., 2016; Gong et al., 2017; Yang et al., 2016; Parikh et al., 2016; Gong et al., 2017; Kim et al., 2019). However, this data-driven technique typically requires large amounts of manual annotation and brings much cost. If large labeled data can’t be obtained, the advantages of deep learning will significantly diminish. To alleviate this problem, active learning is proposed to achieve better performance with fewer labeled training instances (Settles, 2009). Instead of randomly selecting instances, active learning can measure the whole candidate instances according to some criteria, and then select more efficient instances for annotation (Zhang et al., 2017; Shen et al., 2017; Erdmann et al., ; Kasai et al., 2019; Xu et al., 2018). However, previous active learning approaches in natural language processing mainly depend on the entropy-based uncertainty criterion (Settles, 2009), and ignore the characteristics of natural language. To be more specific, if we ignore the linguistic similarity, we may select redundant instances and waste many annotation resources. Thus, how to devise linguistic criteria to measure candidate instances is an important challenge. Recently, pre-trained language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019) have been shown to be powerful for learning language representation. Accordingly, pre-trained language models may provide a reliable way to help capture language characteristics. In this paper, we devise linguistic criteria from a pre-trained language model to capture language characteristics, and then utilize these extra linguistic criteria (noise, coverage and diversity) to enhance active learning. It is shown in Figure 1. Experiments on both English and Chinese sentence matching datasets demonstrate the pre-trained language model can enhance active learning. 2 Methodology In a general active learning scenario, there is a small set of labeled training data P and a large pool of available unlabeled data Q. Active …

Content truncated to 500 words