NTNU-2 at SemEval-2017 Task 10: Identifying Synonym and Hyponym Relations among Keyphrases in Scientific Documents

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 965–968, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics NTNU-2 at SemEval-2017 Task 10: Identifying Synonym and Hyponym Relations among Keyphrases in Scientific Documents Biswanath Barik, Erwin Marsi Department of Computer Science Norwegian University of Science and Technology {biswanath.barik,emarsi}@ntnu.no Abstract This paper presents our relation extraction system for subtask C of SemEval-2017 Task 10: ScienceIE. Assuming that the keyphrases are already annotated in the in- put data, our work explores a wide range of linguistic features, applies various fea- ture selection techniques, optimizes the hyper parameters and class weights and experiments with different problem for- mulations (single classification model vs individual classifiers for each keyphrase type, single-step classifier vs pipeline clas- sifier for hyponym relations). Perfor- mance of five popular classification algo- rithms are evaluated for each problem for- mulation along with feature selection. The best setting achieved an F1 score of 71.0% for synonym and 30.0% for hyponym re- lation on the test data. 1 Problem Description Task C of ScienceIE at SemEval-2017 (Au- genstein et al., 2017) concerns identifying sen- tence level ‘SYNONYM-OF’ (or ‘same-as’) and ‘HYPONYM-OF’ (‘is-a’) relations among three types of keyphrases: PROCESS (PR), TASK (TA) and MATERIAL (MA) in scientific docu- ments. The ‘SYNONYM-OF’ relation is symmet- ric, whereas the ‘HYPONYM-OF’ relation is di- rected. Hyponym relation prediction is thus as- sociated with two ordered subtasks: (1) predict- ing relations between pairs of keyphrases; (2) pre- dicting the direction of the relation. It is assumed that there are no relations between keyphrase of different types. Automatic identification of syn- onym/hyponym relations is useful for many NLP applications, e.g. knowledge base completion and ontology construction. 2 Challenges The relation prediction task of ScienceIE is chal- lenging and quite different from other semantic re- lation prediction task like SemEval-2010 Task 8 (Hendrickx et al., 2009). In SemEval-2010 Task 8, there are two marked nominals in a sentence and the task is to predict if any of nine seman- tic relations hold between the nominal pair. Al- though there are more relations than ScienceIE (9 vs 2), ScienceIE poses different challenges. In- stead of single-word nominals, the keyphrases of ScienceIE are arbitrarily large text spans referring to larger syntactico-semantic units. The top part of Table 1 shows the percentage of keyphrases longer than 10 tokens in the training (10.89%), develop- ment (8.76%) and test (6.71%) data. The problem with such large text spans is to identify features which best represent the keyphrase and contribute most to the relation prediction task. Another challenge of ScienceIE is the occur- rence of multiple keyphrases in one sentence, pro- ducing a large number of possible relations among keyphrase pairs, i.e., n(n−1)/2 for n keyphrases. As most of these are negative instances, the posi- tive and negative classes are imbalanced. A third challenge is the potentially long dis- tance between keyphrase pairs. The middle part of Table 1 shows that there are 49.2%, 57.68% and 43.77% keyphrase pairs …

Content truncated to 500 words