ForceReader: a BERT-based Interactive Machine Reading Comprehension Model with Attention Separation

Proceedings of the 28th International Conference on Computational Linguistics, pages 2676–2686 Barcelona, Spain (Online), December 8-13, 2020 2676 ForceReader: a BERT-based Interactive Machine Reading Comprehension Model with Attention Separation Zheng Chen, Kangjian Wu School of Information and Software Engineering University of Electronic Science and Technology of China, Chengdu, China zchen@uestc.edu.cn, kenjewu@std.uestc.edu.cn Abstract The release of BERT revolutionized the development of NLP. Various BERT-based reading comprehension models have been proposed, thus updating the performance ranking of reading comprehension tasks. However, the above BERT-based models inherently employ BERT’s com- bined input method, representing the input question and paragraph as a single packed sequence, without further modification for reading comprehension. This paper makes an in-depth analysis of this input method, proposes a problem of this approach. We call it attention deconcentration. Accordingly, this paper proposes ForceReader, a BERT-based interactive machine reading com- prehension model. First, ForceReader proposes a novel solution called the Attention Separation Representation to respond to attention deconcentration. Moreover, starting from the logical nature of reading comprehension tasks, ForceReader adopts Multi-mode Reading, and Interactive Rea- soning strategy. For the calculation of attention, ForceReader employs Conditional Background Attention to solve the lack of the overall context semantic after the separation of attention. As an integral model, ForceReader shows a significant improvement in reading comprehension tasks compared to BERT. Moreover, this paper makes detailed visual analyses of the attention and propose strategies accordingly. This may be another argument to the explanations of the attention. 1 Introduction Reading comprehension can be used as a comprehensive test task to evaluate machine understanding of human language(Chen, 2018). Therefore, it has become a primary research topic in natural language pro- cessing. In the past few years, due to new datasets, algorithms, and computing capabilities, deep learning technology has evolved rapidly. Driven by deep learning approaches, machine reading comprehension has made significant breakthroughs. Hermann et al.(2015) of DeepMind proposed CNN/Daily Mail, which known as one of the first large- scale supervised training datasets for machine reading comprehension. They also proposed The Attentive Reader(Hermann et al., 2015), an attention-based Long Short-Term Memory network(Hochreiter and Schmidhuber, 1997) model for reading comprehension. In order to overcome the shortage of CNN/Daily Mail dataset, researchers such as Rajpurkar proposed the SQuAD1.0(Rajpurkar et al., 2016) dataset in 2016, and SQuAD2.0(Rajpurkar et al., 2018) dataset in 2019. These two datasets have rapidly driven a series of studies on deep learning-based machine reading comprehension, resulting in many efficient and ingenious models. Chen et al.(2016) proposed the Stanford Attentive Reader. This end-to-end reading comprehension model combines multi granular language knowledge and attentional mechanisms, become an early representative of neural machine reading comprehension models. Minjoon et al.(2016) proposed a bidirectional attention flow reading model, BiDAF. Based on the BiLSTM backbone network that inherits from Stanford Attentive Reader, BiDAF novelly proposes a bi-directional attention interaction, which is not only from query to context but also from context to query. QANET proposed by Yu et al.(2018) in 2018, adopting a completely different approach. QANET is no longer base on BiLSTM …

Content truncated to 500 words