Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6059–6063, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6059 Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching Bo Shao†‡, Yeyun Gong‡, Weizhen Qi§‡, Nan Duan‡, Xiaola Lin† †Sun Yat-sen University ‡Microsoft Research Asia §University of Science and Technology of China shaobo2@mail2.sysu.edu.cn {yegong, nanduan}@microsoft.com weizhen@mail.ustc.edu.cn linxl@mail.sysu.edu.cn Abstract In this work, we propose an aggregation method to combine the Bidirectional Encoder Representations from Transformer (BERT) with a MatchLSTM layer for Sequence Match- ing. Given a sentence pair, we extract the out- put representations of it from BERT. Then we extend BERT with a MatchLSTM layer to get further interaction of the sentence pair for se- quence matching tasks. Taking natural lan- guage inference as an example, we split BERT output into two parts, which is from premise sentence and hypothesis sentence. At each position of the hypothesis sentence, both the weighted representation of the premise sen- tence and the representation of the current to- ken are fed into LSTM. We jointly train the aggregation layer and pre-trained layer for se- quence matching. We conduct an experiment on two publicly available datasets, WikiQA and SNLI. Experiments show that our model achieves significant improvement compared with state-of-the-art methods on both datasets. 1 Introduction Text sequence matching aims to infer the relation- ship of two text sequences which is a critical prob- lem used in many NLP tasks such as answer se- lection (Yang et al., 2015), natural language infer- ence (Bowman et al., 2015), and paraphrase iden- tification (Dolan et al., 2004). Recent years, various deep learning models have been proposed for text sequence matching tasks (Wang and Jiang, 2016; Tay et al., 2018; Ty- moshenko and Moschitti, 2018; Kim et al., 2018). Most previous works are learned from task spe- cific supervised datasets, which are always limited to the amount, due to the cost of annotation. Re- cently, learning a general representation of words based on language models attracts great attention, and are successfully applied to a range of NLP tasks (McCann et al., 2017; Peters et al., 2018, 2017; Alec Radford, 2018; Devlin et al., 2018). In these works, they first learn general language rep- resentations through training a language model us- ing large scale unlabeled data. Then they use these representations to the downstream tasks. Previous works show that the representations with linguis- tic information perform obvious effectiveness and robustness applied to downstream tasks. In these methods, there are two typical strate- gies using pre-trained models, feature-based (Pe- ters et al., 2017) and fine-tuning approach (De- vlin et al., 2018; Radford et al., 2018). Our model is based on the fine-tuning approach and use BERT (Devlin et al., 2018) as our pre-trained language model. BERT has been directly used in many tasks and achieved significantly improve- ment compared with other methods. However, BERT only uses a standard softmax layer for many different specific …

Content truncated to 500 words