DTSim at SemEval-2016 Task 2: Interpreting Similarity of Texts Based on Automated Chunking, Chunk Alignment and Semantic Relation Prediction

Proceedings of SemEval-2016, pages 809–813, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics DTSim at SemEval-2016 Task 2: Interpreting Similarity of Texts Based on Automated Chunking, Chunk Alignment and Semantic Relation Prediction Rajendra Banjade∗, Nabin Maharjan∗, Nobal B. Niraula, Vasile Rus Department of Computer Science / Institute for Intelligent Systems The University of Memphis Memphis, TN, USA {rbanjade, nmharjan, nbnraula, vrus}@memphis.edu Abstract In this paper we describe our system (DTSim) submitted at SemEval-2016 Task 2: Inter- pretable Semantic Textual Similarity (iSTS). We participated in both gold chunks cate- gory (texts chunked by human experts and provided by the task organizers) and system chunks category (participants had to automat- ically chunk the input texts). We developed a Conditional Random Fields based chunker and applied rules blended with semantic simi- larity methods in order to predict chunk align- ments, alignment types and similarity scores. Our system obtained F1 score up to 0.648 in predicting the chunk alignment types and scores together and was one of the top per- forming systems overall. 1 Introduction Measuring the semantic similarity of texts is to quantify the degree of semantic similarity between a given pair of texts, such as two words or two sen- tences (Rus et al., 2008; Agirre et al., 2015). For example, a similarity score of 0 means that the texts are not similar at all and 5 means that they have same meaning. While useful, such quantitative or even qualitative assessments are hard to interpret be- cause they do not provide details, i.e. they do not explain or justify why the similarity score was as- signed high or low. One way to provide an explana- tory layer to text similarity assessment methods is to align chunks between texts and assigning seman- tic relation to each alignment. To this end, Brock- ett (2007) and Rus et al. (2012) produced datasets ∗* These authors contributed equally to this work where corresponding words (or multiword expres- sions) were aligned and in the latter case their se- mantic relations were explicitly labeled In interpretable Semantic Textual Similarity (iSTS) tasks (Agirre et al., 2016), participants had first to identify the chunks in each sentence (sys chunks) or use the chunks given by the task orga- nizers (gold chunks), and then, align chunks across the two sentences indicating the semantic relation and similarity score of each alignment. The chunk alignment types were EQUI (semantically equiva- lent), OPPO (opposite in meaning), SPE (one chunk is more specific than other), SIMI (similar meanings, but not EQUI, OPPO, SPE), REL (related mean- ings, but not SIMI, EQUI, OPPO, SPE), and NOALI (has no corresponding chunk in the other sentence). The relatedness/similarity scores were assigned in the range of 0 to 5. A pilot on iSTS task was organized in 2015 (Agirre et al., 2015). In 2016, the iSTS allowed many-to-many chunk alignment while in the pilot task of 2015 they only allowed one-to-one or no alignment. Also, a new dataset consisting of stu- dent answers given to a …

Content truncated to 500 words