Combining Shallow and Deep Representations for Text-Pair Classification

Combining Shallow and Deep Representations for Text-Pair Classification Vincent Nguyen1,2 Sarvnaz Karimi1 Zhenchang Xing2 1CSIRO Data61, Sydney, Australia 2The Australian National University, Canberra, Australia {firstname.lastname}@csiro.au {zhenchang.xing}@anu.edu.au Abstract Text-pair classification is the task of determin- ing the class relationship between two sen- tences. It is embedded in several tasks such as paraphrase identification and duplicate ques- tion detection. Contemporary methods use fine-tuned transformer encoder semantic rep- resentations of the classification token in the text-pair sequence from the transformer’s fi- nal layer for class prediction. However, re- search has shown that earlier parts of the network learn shallow features, such as syn- tax and structure, which existing methods do not directly exploit. We propose a novel convolution-based decoder for transformer- based architecture that maximizes the use of encoder hidden features for text-pair classifi- cation. Our model exploits hidden represen- tations within transformer-based architecture. It outperforms a transformer encoder baseline on average by 50% (relative F1-score) on six datasets from the medical, software engineer- ing, and open-domains. Our work shows that transformer-based models can improve text-pair classification by modifying the fine- tuning step to exploit shallow features while improving model generalization, with only a slight reduction in efficiency.1 1 Introduction Text-pair classification determines the class rela- tionship between two sentences; for example, it determines the inference class relationship (entail- ment, contradiction, or neutral) between a premise and a hypothesis (Bowman et al., 2015). Such clas- sification requires interpreting the semantic con- tent of sentences to determine their relationships. Applications of text-pair classification, which we experiment with here, are natural language infer- ence (Bowman et al., 2015), question answering candidate ranking (Ben Abacha et al., 2019) and duplicate question detection (Wang et al., 2019a; 1Code and dataset will be released upon publication. Yang et al., 2019). For these tasks, we consider open, biomedical, and software engineering do- mains. Contemporary methods in text-pair classifica- tion use transformer encoder networks, such as BERT (Devlin et al., 2019), which are popular in natural language processing (Wang et al., 2019b; Sun et al., 2019; Zhu et al., 2019). For these en- coder models, there are several studies improving the model in different aspects. Liu et al. (2019b) and Yang et al. (2019) introduce improvements to the pretraining by removing Next Sentence Pre- diction (NSP) and using a larger batch-size with the LAMB optimizer (You et al., 2020). SciB- ERT (Beltagy et al., 2019) pretrains BERT over publications from SemanticScholar and adjusts the model’s vocabulary to be domain-specific. AL- BERT (Lan et al., 2020) increases model depth, adds layer parameter sharing, and includes the sen- tence coherence over NSP. These studies primarily improve the transformer architecture by scaling up overall model capacity through dataset source (Lee et al., 2019; Alsentzer et al., 2019) and adjustment of the pretraining task (Lan et al., 2020; Joshi et al., 2019). They use only the classification token as the primary fea- ture for classification. These improvements also require pretraining the architecture, which requires a high amount of computing resources. Increasing model size and …

Content truncated to 500 words