MIT-MEDG at SemEval-2018 Task 7: Semantic Relation Classification via Convolution Neural Network

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 798–804 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics MIT-MEDG at SemEval-2018 Task 7: Semantic Relation Classification via Convolution Neural Network Di Jin MIT CSAIL Cambridge, MA jindi15@mit.edu Franck Dernoncourt Adobe Research San Jose, CA dernonco@adobe.com Elena Sergeeva MIT CSAIL Cambridge, MA elenaser@mit.edu Matthew B. A. McDermott MIT CSAIL Cambridge, MA mmd@mit.edu Geeticka Chauhan MIT CSAIL Cambridge, MA geeticka@mit.edu Abstract SemEval 2018 Task 7 tasked participants to build a system to classify two entities within a sentence into one of the 6 possible relation types. We tested 3 classes of models: Linear classifiers, Long Short-Term Memory (LSTM) models, and Convolutional Neural Network (CNN) models. Ultimately, the CNN model class proved most performant, so we special- ized to this model for our final submissions. We improved performance beyond a vanilla CNN by including a variant of negative sam- pling, using custom word embeddings learned over a corpus of ACL articles, training over corpora of both tasks 1.1 and 1.2, using re- versed feature, using part of context words beyond the entity pairs and using ensem- ble methods to improve our final predictions. We also tested attention based pooling, up- sampling, and data augmentation, but none improved performance. Our model achieved rank 6 out of 28 (macro-averaged F1-score: 72.7) in subtask 1.1, and rank 4 out of 20 (macro F1: 80.6) in subtask 1.2. 1 Introduction SemEval 2018 Task 7 (Gbor et al., 2018) focuses on relation classification and extraction on a cor- pus of 350 scientific paper abstracts consisting of 1228 and 1248 annotated sentences for subtasks 1.1 and 1.2, respectively. There are six possible relations: USAGE, RESULT, MODEL-FEATURE, PART WHOLE, TOPIC, and COMPARE. Given this data, our task is to take an exam- ple sentence, as well as the left and right enti- ties within that sentence, and an indicator as to whether the relation is reversed, and predict the relation type for that sentence. In subtasks 1.1 and 1.2, all presented sentences have a relation. We submitted predictions based on a self- ensembled convolutional neural network (CNN) model trained with a negative sampling aug- mented loss using ACL-specific embeddings as input features. We achieved rank 6 out of 28 (macro-averaged F1-score: 72.7) in subtask 1.1, and rank 4 out of 20 (macro F1 80.6) in sub- task 1.2. 2 Related Work Previous SemEval challenges have explored re- lation identification and extraction. The 2010 SemEval Task 8 (Hendrickx et al., 2010) explored classification of natural language relations, such as CONTENT-CONTAINER or ENTITY-ORIGIN. This challenge differs from ours in its generaliz- ability; our relations are specific to ACL papers (e.g. MODEL-FEATURE) whereas the 2010 rela- tions are more general, and may necessitate more common-sense knowledge than the 2018 relations. The 2010 data has been extensively studied and has offered significant opportunity for other re- searchers to test their model. Rink and Harabagiu (2010) produced a strong SVM/LR model to at- tack this challenge. Several deep architectures have …

Content truncated to 500 words