Sentence Matching with Syntax- and Semantics-Aware BERT

Proceedings of the 28th International Conference on Computational Linguistics, pages 3302–3312 Barcelona, Spain (Online), December 8-13, 2020 3302 Sentence Matching with Syntax- and Semantics-Aware BERT Tao Liu1, Xin Wang2, Chengguo Lv1, Ranran Zhen1 and Guohong Fu3∗ 1. School of Computer Science and Technology, Heilongjiang University, China 2. CuraCloud Corporation, Seattle, WA, USA 3. Institute of Artificial Intelligence, Soochow University, China taoooo1009@gmail.com, xinw@curacloudcorp.com 2004085@hlju.edu.cn, zenrran@gmail.com ghfu@suda.edu.cn Abstract Sentence matching aims to identify the special relationship between two sentences, and plays a key role in many natural language processing tasks. However, previous studies mainly focused on exploiting either syntactic or semantic information for sentence matching, and no studies con- sider integrating both of them. In this study, we propose integrating syntax and semantics into BERT with sentence matching. In particular, we use an implicit syntax and semantics integration method that is less sensitive to the output structure information. Thus the implicit integration can alleviate the error propagation problem. The experimental results show that our approach has achieved state-of-the-art or competitive performance on several sentence matching datasets, demonstrating the benefits of implicitly integrating syntactic and semantic features in sentence matching. 1 Introduction Sentence matching aims to determine the specific relationship between two sentences. It plays a key role in many natural language processing tasks such as natural language inference (Bowman et al., 2015), paraphrase identification (Wang et al., 2017), and answer selection (Yang et al., 2015). Learning contextual representations is fundamental to many sentence matching tasks. Earlier works explore contextual features of sentences individually in a neural network classifier to predict sentence relationship (Bowman et al., 2015; Chen et al., 2017a). However, they do not consider the more fine- grained comparison information such as word-to-word correspondence between sentences. Recent stud- ies achieve better results by taking into account contextual features and fine-grained interaction between sentences at the same time (Tay et al., 2018a; Kim et al., 2019; Yang et al., 2019a). More recently, with the development of pre-training language models (Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b), substantial progress has been made in sentence matching tasks. For example, BERT (Bidirec- tional Encoder Representations from Transformers) (Devlin et al., 2019) has proven to be a powerful representation of contexts for sentence relationship prediction, in that BERT representations can implic- itly capture a rich hierarchy of linguistic information within sentences (Jawahar et al., 2019). In addition to contextual representation, structural information such as syntactic structures and seman- tic roles has shown to be effective to enhance sentence matching tasks and other NLP systems (Chen et al., 2017a; Zhang et al., 2020). On the one hand, with the maturity of dependency parsing and semantic role labeling technologies in recent years, it is relatively easier to obtain reliable structural information for sentence matching. On the other hand, integrating structural information with pre-training language models has been a promising way to enhance sentence matching performance. Semantic role labeling (SRL) is expressed as predicate-argument relationships to explore who did what to whom, when and …

Content truncated to 500 words