TCDSCSS: Dimensionality Reduction to Evaluate Texts of Varying Lengths - an IR Approach

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 619–623, Dublin, Ireland, August 23-24, 2014. TCDSCSS: Dimensionality Reduction to Evaluate Texts of Varying Lengths - an IR Approach Arun Jayapal Dept of Computer Science Trinity College Dublin jayapala@cs.tcd.ie Martin Emms Dept of Computer Science Trinity College Dublin martin.emms@cs.tcd.ie John D.Kelleher School of Computing Dublin Institute of Technology john.d.kelleher@dit.ie Abstract This paper provides system description of the cross-level semantic similarity task for the SEMEVAL-2014 workshop. Cross- level semantic similarity measures the de- gree of relatedness between texts of vary- ing lengths such as Paragraph to Sen- tence and Sentence to Phrase. Latent Se- mantic Analysis was used to evaluate the cross-level semantic relatedness between the texts to achieve above baseline scores, tested on the training and test datasets. We also tried using a bag-of-vectors approach to evaluate the semantic relatedness. This bag-of-vectors approach however did not produced encouraging results. 1 Introduction Semantic relatedness between texts have been dealt with in multiple situations earlier. But it is not usual to measure the semantic relatedness of texts of varying lengths such as Paragraph to Sen- tence (P2S) and Sentence to Phrase (S2P). This task will be useful in natural language process- ing applications such as paraphrasing and summa- rization. The working principle of information re- trieval system is the motivation for this task, where the queries are not of equal lengths compared to the documents in the index. We attempted two ways to measure the semantic similarity for P2S and S2P in a scale of 0 to 4, 4 meaning both texts are similar and 0 being dissimilar. The first one is Latent Semantic Analysis (LSA) and second, a bag-of-vecors (BV) approach. An example of tar- get similarity ratings for comparison type S2P is provided in table 1. This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence de- tails: http://creativecommons.org/licenses/ by/4.0/ Sentence: Schumacher was undoubtedly one of the very greatest racing drivers there has ever been, a man who was routinely, on every lap, able to dance on a limit accessible to almost no-one else. Score Phrase 4 the unparalleled greatness of Schumachers driving abilities 3 driving abilities 2 formula one racing 1 north-south highway 0 orthodontic insurance Table 1: An Example - Sentence to Phrase simi- larity ratings for each scale 2 Data The task organizers provided training data, which included 500 pairs of P2S, S2P, Phrase to Word (P2W) and their similarity scores. The training data for P2S and S2P included text from different genres such as Newswire, Travel, Metaphoric and Reviews. In the training data for P2S, newswire text constituted 36% of the data, while reviews constituted 10% of the data and rest of the three genres shared 54% of the data. Considering the different genres provided in the training data, a chunk of data provided for NIST TAC’s Knowledge Base Population was used for building a term-by-document matrix on which …

Content truncated to 500 words