UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement

Proceedings of SemEval-2016, pages 1103–1108, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement Hua He1, John Wieting2, Kevin Gimpel2, Jinfeng Rao1, and Jimmy Lin3 1 Department of Computer Science, University of Maryland, College Park 2 Toyota Technological Institute at Chicago 3 David R. Cheriton School of Computer Science, University of Waterloo {huah,jinfeng}@umd.edu, {jwieting,kgimpel}@ttic.edu, jimmylin@uwaterloo.ca Abstract We describe an attention-based convolutional neural network for the English semantic tex- tual similarity (STS) task in the SemEval- 2016 competition (Agirre et al., 2016). We develop an attention-based input interaction layer and incorporate it into our multi- perspective convolutional neural network (He et al., 2015), using the PARAGRAM-PHRASE word embeddings (Wieting et al., 2016) trained on paraphrase pairs. Without using any sparse features, our final model outperforms the winning entry in STS2015 when evaluated on the STS2015 data. 1 Introduction Measuring the semantic textual similarity (STS) of two pieces of text remains a fundamental problem in language research. It lies at the core of many language processing tasks, including paraphrase de- tection (Xu et al., 2014), question answering (Lin, 2007), and query ranking (Duh, 2009). The STS problem can be formalized as: given a query sentence S1 and a comparison sentence S2, the task is to compute their semantic similar- ity in terms of a similarity score sim(S1, S2). The SemEval Semantic Textual Similarity tasks (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015; Agirre et al., 2016) are a popu- lar evaluation venue for the STS problem. Over the years the competitions have made more than 15, 000 human annotated sentence pairs publicly available, and have evaluated over 300 system runs. Traditional approaches are based on hand-crafted feature engineering (Wan et al., 2006; Madnani et al., 2012; Fellbaum, 1998; Fern and Stevenson, 2008; Das and Smith, 2009; Guo and Diab, 2012; Sultan et al., 2014; Kashyap et al., 2014; Lynum et al., 2014). Competitive systems in recent years are mostly based on neural networks (He et al., 2015; Tai et al., 2015; Yin and Sch¨utze, 2015; He and Lin, 2016), which can alleviate data sparseness with pre- training and distributed representations. In this paper, we extend the multi-perspective convolutional neural network (MPCNN) of He et al. (2015). Most previous neural network models, including the MPCNN, treat input sentences sepa- rately, and largely ignore context-sensitive interac- tions between the input sentences. We address this problem by utilizing an attention mechanism (Bah- danau et al., 2014) to develop an attention-based in- put interaction layer (Sec. 3). It converts the two in- dependent input sentences into an inter-related sen- tence pair, which can help the model identify im- portant input words for improved similarity mea- surement. We also use the strongly-performing PARAGRAM-PHRASE word embeddings (Wieting et al., 2016) (Sec. 4) trained on phrase pairs from the Paraphrase Database (Ganitkevitch et al., 2013). These components comprise our submission to the SemEval-2016 STS competition …

Content truncated to 500 words