Learning the Impact and Behavior of Syntactic Structure: A Case Study in Semantic Textual Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 688–696, Hissar, Bulgaria, Sep 7–9 2015. Learning the Impact and Behavior of Syntactic Structure: A Case Study in Semantic Textual Similarity Ngoc Phuoc An Vo University of Trento, Fondazione Bruno Kessler Trento, Italy ngoc@fbk.eu Octavian Popescu IBM Research, T.J. Watson Yorktown, US o.popescu@us.ibm.com Abstract We present a case study on the role of syn- tactic structures towards resolving the Se- mantic Textual Similarity (STS) task. Al- though various approaches have been pro- posed, the research of using syntactic in- formation to determine the semantic simi- larity is a relatively under-researched area. At the level of syntactic structure, it is not clear how significant the syntactic struc- ture contributes to the overall accuracy of the task. In this paper, we analyze the impact of syntactic structure towards the overall performance and its behavior in different score ranges of the STS seman- tic scale. 1 Introduction The task Semantic Textual Similarity (STS) has become a noticed trend in the Natural Language Processing (NLP) community since the SemEval 2012 with a large number of participating sys- tems.1 The participating systems should be able to determine the degree of similarity for pair of short pieces of text, like sentences, where the similarity score is normally obtained by averaging the opin- ion of several annotators. A semantic similarity score is usually a real number in a semantic scale [0-5], from no relevance to semantic equivalence. Some examples from the STS 2012 dataset with associated similarity scores (by human judgment) are as below: _ In May 2010, the troops attempted to invade Kabul. vs. The US army invaded Kabul on May 7th last year, 2010. (score = 4.0) _ Vivendi shares closed 3.8 percent up in Paris at 15.78 euros. vs. Vivendi shares were 0.3 percent 1http://www.cs.york.ac.uk/semeval-2012/task6 up at 15.62 euros in Paris at 0841 GMT. (score = 2.6) _ The woman is playing the violin. vs. The young lady enjoys listening to the guitar. (score = 1.0) The literature (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014) shows that in order to compute the semantic similarity, most STS sys- tems rely on pairwise similarity, either using tax- onomies (WordNet (Fellbaum, 1998)) or distribu- tional semantic models LDA (Blei et al., 2003), LSA (Landauer et al., 1998), ESA (Gabrilovich and Markovitch, 2007), and word/n-grams over- lap as main features to train supervised models, or deploy unsupervised word-alignment metrics to align two given texts. In common sense, syntactic structure may keep a crucial part for human being to understand the meaning of a given text. Thus, it also may help to identify the semantic equivalence between two given texts. However, in the STS task, very few systems provide evidence of the contribution of syntactic structure in its overall performance. Fol- lowing the work in the literature (Vo and Popescu, 2015), we would like to make a deeper study and analysis whose contribution consists of two folds, on the STS 2012, 2013, and 2014 datasets …

Content truncated to 500 words