DABERT: Dual Attention Enhanced BERT for Semantic Matching

Proceedings of the 29th International Conference on Computational Linguistics, pages 1645–1654 October 12–17, 2022. 1645 DABERT: Dual Attention Enhanced BERT for Semantic Matching Sirui Wang1,* , Di Liang1,2,† , Jian Song† , Yuntao Li †, Wei Wu† Tsinghua University, Beijing, China* Meituan Inc.,Beijing, China† {wangsirui,liangdi04,songjian20,liyuntao04,wuwei30}@meituan.com Abstract Transformer-based pre-trained language mod- els such as BERT have achieved remarkable re- sults in Semantic Sentence Matching. However, existing models still suffer from insufficient ability to capture subtle differences. Minor noise like word addition, deletion, and mod- ification of sentences may cause flipped pre- dictions. To alleviate this problem, we pro- pose a novel Dual Attention Enhanced BERT (DABERT) to enhance the ability of BERT to capture fine-grained differences in sentence pairs. DABERT comprises (1) Dual Attention module, which measures soft word matches by introducing a new dual channel alignment mechanism to model affinity and difference at- tention. (2) Adaptive Fusion module, this mod- ule uses attention to learn the aggregation of difference and affinity features, and generates a vector describing the matching details of sen- tence pairs. We conduct extensive experiments on well-studied semantic matching and robust- ness test datasets, and the experimental results show the effectiveness of our proposed method. 1 Introduction Semantic Sentence Matching (SSM) is a funda- mental NLP task. The goal of SSM is to compare two sentences and identify their semantic relation- ship. In paraphrase identification, SSM is used to determine whether two sentences are paraphrase or not (Madnani et al., 2012). In natural language inference task, SSM is utilized to judge whether a hypothesis sentence can be inferred from a premise sentence (Bowman et al., 2015). In the answer sen- tence selection task, SSM is employed to assess the relevance between query-answer pairs and rank all candidate answers (Wang et al., 2020). Across the rich history of semantic sentence matching research, there have been two main streams of studies for solving this problem. One 1These authors contributed equally to this work. 2Corresponding author. The secretaries knew the students. The secretaries knew the students slept . Is a girl looks at you, someone stared at you ? what does it meant ? What does it mean when S1: S2: S1: S2: Is the weather sunny this last weekend ? Is the weather sunny weekend ? S1: S2: S2: S1: S2: S2: S1: S2: S2: S1: S2: S2: S1: S2: S2: Figure 1: Example sentences with similar text but dif- ferent semantics. S1 and S2 are sentence pair. is to utilize a sentence encoder to convert sen- tences into low-dimensional vectors in the latent space, and apply a parameterized function to learn the matching scores between them (Reimers and Gurevych, 2019; Wang et al., 2020). Another paradigm adopts attention mechanism to calculate scores between tokens from two sentences, and then the matching scores are aggregated to make a sentence-level decision (Chen et al., 2016; Tay et al., 2017). In recent years, pre-trained models, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), have became much more …

Content truncated to 500 words