UTA DLNLP at SemEval-2016 Task 1: Semantic Textual Similarity: A Unified Framework for Semantic Processing and Evaluation

Proceedings of SemEval-2016, pages 584–587, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics UTA DLNLP at SemEval-2016 Task 1: Semantic Textual Similarity: A Unified Framework for Semantic Processing and Evaluation Peng Li Computer Science and Engineering University of Texas at Arlington jerryli1981@gmail.com Heng Huang∗ Computer Science and Engineering University of Texas at Arlington heng@uta.edu Abstract In this paper, we propose a deep neural net- work based natural language processing sys- tem for semantic textual similarity prediction. We leverage multi-layer bidirectional LSTM to learn sentence representation. After that, we construct matching features followed by High- way Multilayer Perceptron to make predic- tions. Experimental results demonstrate that this approach can’t get better results on stan- dard evaluation datasets. 1 Introduction Traditional approaches (Lai and Hockenmaier, 2014; Zhao et al., 2014; Jimenez et al., 2014) for semantic textual similarity prediction usually build the supervised model using a variety of hand crafted features. Hundreds of features generated at differ- ent linguistic levels are exploited to boost classifica- tion. With the success of deep learning in many ma- chine learning related applications, there has been much interest in applying deep neural network based techniques to further improve the prediction tasks in natural language processing (NLP) (Socher et al., 2011b; Iyyer et al., 2014; Tai et al., 2015). A key component of deep neural network is word embeddings which serves as a lookup table to get word representations. From low-level NLP tasks such as language modeling, POS tagging, name en- tity recognition, and semantic role labeling, to high- ∗To whom all correspondence should be addressed. This work was partially supported by NSF-IIS 1117965, NSF-IIS 1302675, NSF-IIS 1344152, NSF-DBI 1356628, NIH R01 AG049371. level tasks such as machine translation, informa- tion retrieval and semantic analysis (Kalchbrenner and Blunsom, 2013; Socher et al., 2011a; Tai et al., 2015). Deep word representation learning has demonstrated its importance for these tasks. All the tasks get performance improvement via further learning either word level representations or sen- tence level representations. In this work, we focus on deep neural network based semantic textual similarity prediction. We use multi-layer bidirectional LSTM (Long Short Term Memory) (Graves et al., 2013) to learn sentence rep- resentations. After that, we construct matching fea- tures followed by Highway Multilayer Perceptron to learn high-level hidden matching feature representa- tions. Below, we will briefly introduce Multi-Layer Bidirectional LSTM. 2 Multi-Layer Bidirectional LSTM 2.1 RNN vs LSTM Recurrent neural networks (RNNs) are capable of modeling sequences of varying lengths via the re- cursive application of a transition function on a hid- den state. For example, at each time step t, an RNN takes the input vector xt ∈ Rn and the hidden state vector ht−1 ∈ Rm, then applies affine transforma- tion followed by an element-wise nonlinearity such as hyperbolic tangent function to produce the next hidden state vector ht: ht = tanh(Wxt + Uht−1 + b) . (1) A major issue of RNNs using these transition functions is that it is difficult to learn long-range de- …

Content truncated to 500 words