Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 108–118 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 108 Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension Matthias Blohm, Glorianna Jagfeld, Ekta Sood, Xiang Yu, Ngoc Thang Vu Institute for Natural Language Processing (IMS) Universit¨at Stuttgart, Germany {blohmms,jagfelga,soodea,xiangyu,thangvu} @ims.uni-stuttgart.de Abstract We propose a machine reading comprehension model based on the compare-aggregate frame- work with two-staged attention that achieves state-of-the-art results on the MovieQA ques- tion answering dataset. To investigate the lim- itations of our model as well as the behavioral difference between convolutional and recur- rent neural networks, we generate adversarial examples to confuse the model and compare to human performance. Furthermore, we assess the generalizability of our model by analyz- ing its differences to human inference, draw- ing upon insights from cognitive science. 1 Introduction Current state-of-the-art deep learning (DL) mod- els outperform other techniques in many tasks including computer vision (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) and more recently natural language processing (NLP) (Collobert et al., 2011). Neural-based NLP systems often use word embeddings (Ben- gio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013) which are then fed into a convolutional neural network (CNN) (LeCun et al., 1990; Waibel et al., 1990) or a recurrent neural network (RNN) (Elman, 1990; Hochre- iter and Schmidhuber, 1997) for further classifica- tion. These approaches proved to be successful for many NLP tasks (Mikolov et al., 2010; Kim, 2014; Hu et al., 2014; Bahdanau et al., 2014). Along with the success of DL in a wide range of appli- cations, adversarial examples (Goodfellow et al., 2014) - that aim to confuse the system - have gained popularity in a wide range of research com- munities such as computer vision and NLP, since they can reveal the limitations in the generalizabil- ity of the models. As opposed to adversarial ex- amples in computer vision, which are computed on continuous data and can thus easily be imper- ceptible if desired, adversarial attacks in NLP en- tail the necessity to perform discrete and percep- tible changes to the data. Thus, attack methods for computer vision such as the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) cannot be directly applied to NLP. Machine comprehension has recently received increased interest in the NLP community (Yang et al., 2015; Tapaswi et al., 2016; Rajpurkar et al., 2016; Chen et al., 2016). Neural network models perform reasonably well on many data sets with different question answering setups, e.g. multi- ple choice or answer generation (Wang and Jiang, 2016; Liu et al., 2017; Yu et al., 2018). Among others, Wang and Jiang (2016) pro- posed the compare-aggregate framework, which uses an attention mechanism (Luong et al., 2015) to compare the question and candidate answers, and a CNN to aggregate information. However, there is still an ongoing debate whether CNNs or RNNs are more suitable to NLP …

Content truncated to 500 words