Relying on Discourse Analysis to Answer Complex Questions by Neural Machine Reading Comprehension

Proceedings of Recent Advances in Natural Language Processing, pages 444–453 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_051 444 Relying on Discourse Analysis to Answer Complex Questions by Neural Machine Reading Comprehension Boris Galitsky Oracle Inc. / Redwood Shores, CA, USA boris.galitsky@oracle.com Dmitry Ilvovsky National Research University Higher School of Economics / Moscow, Russia dilvovsky@hse.ru Elizaveta Goncharova National Research University Higher School of Economics / Moscow, Russia egoncharova@hse.ru Abstract Machine reading comprehension (MRC) is one of the most challenging tasks in natural language processing domain. Recent state-of- the-art results for MRC have been achieved with the pre-trained language models, such as BERT and its modifications. Despite the high performance of these models, they still suffer from the inability to retrieve correct answers from the detailed and lengthy passages. In this work, we introduce a novel scheme for in- corporating the discourse structure of the text into a self-attention network, and, thus, en- rich the embedding obtained from the stan- dard BERT encoder with the additional linguis- tic knowledge. We also investigate the influ- ence of different types of linguistic informa- tion on the model’s ability to answer complex questions that require deep understanding of the whole text. Experiments performed on the SQuAD benchmark and more complex ques- tion answering datasets have shown that lin- guistic enhancing boosts the performance of the standard BERT model significantly. 1 Introduction Machine reading comprehension (MRC) reflects the ability to read and understand an unstructured text and answer questions regarding it. Aiming to find the relevant answer to a question in the form of a text span, the MRC models should demon- strate deep understanding of the language and text organization. Transformer models that achieve state-of-the- art results on multiple natural language process- ing (NLP) tasks have been successfully applied to the MRC. However, while the ideal MRC model should read most words superficially and pay atten- tion only to the essential ones (Wang et al., 2017), the attention mechanism in the standard transform- ers attends to all words without explicit constraint which results in inaccurate concentration on some less important text spans. Lately, the researchers have actively examined the ability of the deep learn- ing (DL) models to understand language and build accurate linguistic-enhanced internal representa- tion. Recent works have revealed that traditional DL models that ignore additional linguistic knowledge, such as syntax or semantic, achieves lower accu- racy on such complex tasks as natural language understanding (NLU) or MRC (Roth and Lapata, 2016; Marcheggiani and Titov, 2017; He et al., 2018). It has been shown that incorporating explicit syntactic (Hu et al., 2019) and semantic (Zhang et al., 2020a) relations into the attention mecha- nism leads to better linguistically motivated word representations beneficial for the MRC task. Moreover, providing exact, concise answers fre- quently requires not just syntactic/meaning similar- ity but an overall structure of thoughts expressed by an author (Galitsky et al., 2013), i.e., some claims introduced by an author and logical connections existing among them. This information is encoded by discourse structure of a text …

Content truncated to 500 words