Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering

Proceedings of the 27th International Conference on Computational Linguistics, pages 3890–3902 Santa Fe, New Mexico, USA, August 20-26, 2018. 3890 Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering Wuwei Lan and Wei Xu Department of Computer Science and Engineering Ohio State University {lan.105, xu.1265}@osu.edu Abstract In this paper, we analyze several neural network designs (and their variations) for sentence pair modeling and compare their performance extensively across eight datasets, including paraphrase identification, semantic textual similarity, natural language inference, and question answering tasks. Although most of these models have claimed state-of-the-art performance, the original papers often reported on only one or two selected datasets. We provide a systematic study and show that (i) en- coding contextual information by LSTM and inter-sentence interactions are critical, (ii) Tree-LSTM does not help as much as previously claimed but surprisingly improves performance on Twitter datasets, (iii) the Enhanced Sequential Inference Model (Chen et al., 2017) is the best so far for larger datasets, while the Pairwise Word Interaction Model (He and Lin, 2016) achieves the best performance when less data is available. We release our implementations as an open-source toolkit. 1 Introduction Sentence pair modeling is a fundamental technique underlying many NLP tasks, including the following: • Semantic Textual Similarity (STS), which measures the degree of equivalence in the underlying se- mantics of paired snippets of text (Agirre et al., 2016). • Paraphrase Identification (PI), which identifies whether two sentences express the same meaning (Dolan and Brockett, 2005; Xu et al., 2015). • Natural Language Inference (NLI), also known as recognizing textual entailment (RTE), which con- cerns whether a hypothesis can be inferred from a premise, requiring understanding of the semantic similarity between the hypothesis and the premise (Dagan et al., 2006; Bowman et al., 2015). • Question Answering (QA), which can be approximated as ranking candidate answer sentences or phrases based on their similarity to the original question (Yang et al., 2015). • Machine Comprehension (MC), which requires sentence matching between a passage and a question, pointing out the text region that contains the answer. (Rajpurkar et al., 2016). Traditionally, researchers had to develop different methods specific for each task. Now neural networks can perform all the above tasks with the same architecture by training end to end. Various neural models (He and Lin, 2016; Chen et al., 2017; Parikh et al., 2016; Wieting et al., 2016; Tomar et al., 2017; Wang et al., 2017; Shen et al., 2017a; Yin et al., 2016) have declared state-of-the-art results for sentence pair modeling tasks; however, they were carefully designed and evaluated on selected (often one or two) datasets that can demonstrate the superiority of the model. The research questions are as follows: Do they perform well on other tasks and datasets? How much performance gain is due to certain system design choices and hyperparameter optimizations? This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 3891 To answer these questions and better understand …

Content truncated to 500 words