YNU_AI1799 at SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge of Different model ensemble

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1038–1042 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics YNU AI1799 at SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge of Different model ensemble QingXun Liu, HongDou Yao, Xiaobing Zhou∗, Ge Xie School of Information Science and Engineering Yunnan University, Yunnan, P.R. China ∗Corresponding author, zhouxb.cn@gmail.com Abstract In this paper, we describe a machine read- ing comprehension system that participated in SemEval-2018 Task 11: Machine Compre- hension using commonsense knowledge. In this work, we train a series of neural network models such as multi-LSTM, BiLSTM, multi- BiLSTM-CNN and attention-based BiLSTM, etc. On top of some sub models, there are two kinds of word embedding: (a) general word embedding generated from unsupervised neu- ral language model; and (b) position embed- ding generated from general word embedding. Finally, we make a hard vote on the predictions of these models and achieve relatively good result. The proposed approach achieves 8th place in Task 11 with the accuracy of 0.7213. 1 Introduction Machine Comprehension using Commonsense Knowledge is a well-researched problem in NLP. In order to simplify the task of the process, we turn this task into text classification work and use a deep learning neural network to fulfill it. The method of deep learning models used in text analysis has achieved numerous notable ad- vances in recently years (e.g., (Breck et al., 2007), (Yessenalina and Cardie, 2011) and (Socher et al., 2011)). However, in most previous works, the tasks are to apply a single model to a particular data set task. The single model is a vertical stack of multiple hidden layers, which is not good for text analy- sis and processing. The first drawback is the need to consume more hardware resources, followed by over-fitting and loss of feature information. So the task here is to apply different structure sub models to the same train-set. We train many classic sub models with one layer on top of word embedding, like LSTM, CNN, Attention,Attention+BiLSTM, multi-BiLSTM+CNN and some other models which are slightly different from the above mod- els with different activation functions and differ- ent layers inside the model. In each single model, we use a flag to determine which embedding tool is used or not. Most of the deep learning involve word vectors represented by neural language models ((Morin and Bengio, 2005) , (Yih et al., 2011) and (Mikolov et al., 2013)). Using the learned word vectors for classification task will naturally in- crease the effect. Word vectors are expressed as a hidden-layer word vector of the specified di- mension (1-of-V, here V is the vocabulary size), the training methods can be found here1. In our system, we introduce different word vectors: 100 billion words of Google News, Glove vectors of 100 dimensions and word vectors self-trained on the basis of official task data. Finally, a hard vote(the majority voting from result document) is made on the results of those different models. Many tasks often suffer from insufficient …

Content truncated to 500 words