A Multi-Attention based Neural Network with External Knowledge for Story Ending Predicting Task

Proceedings of the 27th International Conference on Computational Linguistics, pages 1754–1762 Santa Fe, New Mexico, USA, August 20-26, 2018. 1754 A Multi-Attention based Neural Network with External Knowledge for Story Ending Predicting Task Qian Li1, Ziwei Li1, Jin-Mao Wei1, Yanhui Gu2, Adam Jatowt3, Zhenglu Yang1 CCCE, Nankai University,China1 School of CS and Technology, Nanjing Normal University, China2 Graduate School of Informatics, Kyoto University, Japan3 {liqian515,lzw nku}@mail.nankai.edu.cn, weijm@nankai.edu.cn, gu@jnu.edu.cn, adam@dl.kuis.kyoto-u.ac.jp, yangzl@nankai.edu.cn Abstract Enabling a mechanism to understand a temporal story and predict its ending is an interesting issue that has attracted considerable attention, as in case of the ROC Story Cloze Task (SCT). In this paper, we develop a multi-attention based neural network (MANN) with well-designed optimizations, like Highway Network, and concatenated features with embedding representations into the hierarchical neural network model. Considering the particulars of the specific task, we thoughtfully extend MANN with external knowledge resources, exceeding state-of-the-art results obviously. Furthermore, we develop a thorough understanding of our model through a careful hand analysis on a subset of the stories. We identify what traits of MANN contribute to its outperformance and how external knowledge is obtained in such an ending prediction task. 1 Introduction The prediction on story endings is an important and interesting application because it is involved with several essential issues, such as textual semantic understanding, logical reasoning and natural text gen- eration. Most previous studies on the subject of common sense story understanding mainly focus on generating guesses for a missing event, such as matching explicit information in a given context (Cham- bers and Jurafsky, 2008), paying attention to specific types of common sense knowledge, like event schema (Chambers and Jurafsky, 2009), or concentrating on unsupervised learning (Chambers and Ju- rafsky, 2008). Although numerous studies have addressed the issue, training machines to be able to understand underlying narrative structures is still a challenging task. Previous research is limited at the shallow technique requirement of evaluation and noisy knowledge resources. To facilitate the evaluation and benchmark the problem in the literature, the Story Cloze Task (SCT) has been introduced to predict what should be the “right” ending to a story (Mostafazadeh et al., 2016), which consists of daily events. The common strategies utilized to perform the task can be classified into two kinds of approaches: (1) traditional machine learning techniques with optimal feature engineering; and (2) deep learning-based models with effective strategies (Cai et al., 2017). The task is published with an unlabeled training set that consists of one-correct-ending stories, which is a notable impediment for further research. Some studies investigated fake ending generation, which obtained far more satisfying results. Most supervised learning approaches are trained from the finite evaluation set, ignoring the sheer volume of training corpus. Understanding daily stories requires not only common sense experience sharing, but also a thorough understanding of text learned from common sense knowledge resources. We propose an effective multi- attention based neural network (MANN) and broaden our model with external knowledge. The model is superior in three aspects: (i) it adds …

Content truncated to 500 words