Does Structure Matter? Encoding Documents for Machine Reading Comprehension

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4626–4634 June 6–11, 2021. ©2021 Association for Computational Linguistics 4626 Does Structure Matter? Encoding Documents for Machine Reading Comprehension Hui Wan Song Feng Chulaka Gunasekara Siva Sankalp Patel Sachindra Joshi Luis A. Lastras IBM Research AI {hwan@us, sfeng@us, chulaka.gunasekara@}.ibm.com {siva.sankalp.patel@, jsachind@in, lastrasl@us}.ibm.com Abstract Machine reading comprehension is a chal- lenging task especially for querying docu- ments with deep and interconnected contexts. Transformer-based methods have shown ad- vanced performances on this task; however, most of them still treat documents as a flat se- quence of tokens. This work proposes a new Transformer-based method that reads a docu- ment as tree slices. It contains two modules for identifying more relevant text passage and the best answer span respectively, which are not only jointly trained but also jointly con- sulted at inference time. Our evaluation re- sults show that our proposed method outper- forms several competitive baseline approaches on two datasets from varied domains. 1 Introduction Machine Reading Comprehension (MRC) is the task of reading a given text and answering ques- tions about it (Liu et al., 2019). Some MRC tasks such as SQuAD (Rajpurkar et al., 2016, 2018),and ShARC (Saeidi et al., 2018) provide a short text snippets as the context documents; while others such as TriviaQA (Joshi et al., 2017), Natural Ques- tions (Kwiatkowski et al., 2019) and Doc2Dial (Feng et al., 2020) use full articles as documents. Most top performing models on MRC tasks use different variants of Transformers (Vaswani et al., 2017). Transformer-based models typically only consider a certain number of tokens, utilize a slid- ing window approach (Richardson et al., 2013) or segment the document into passages (Hu et al., 2019; Wang et al., 2019) due to the constraint on the size of input sequence. More recent works ex- plore how to scale up input length (Yang et al., 2019; Beltagy et al., 2020; Kitaev et al., 2020; Wang et al., 2020; Ainslie et al., 2020) but still mainly focus on flat sequences. In addition to scal- ing up input length, ETC(Ainslie et al., 2020) also propose to deal with encoding structured inputs How to appeal a TVB ticket You can appeal online if you have been convicted in a NYSTVB the TVB traffic ticket number full name, date of birth, and gender ... U1: I prefer not to deal with mail. How else can I appeal my ticket please? A1: You can appeal online. U2: What do I need? A2: You will need TVB traffic ticket No., full name,... By Mail the conviction occurred within the last 30 days At a TVB office Online You will need Tree View ... ... ... ... <h2>How to appeal a TVB ticket</h2> <h3>When can appeal a TVB ... </h3> <p>You must file your appeal within 30 days... </p> <h3>At a TVB office</h3> <p> Sorry you cannot submit at a TVB office ... </p> <h3>By Mail</h3> ... <h3>Online </h3> <h4>You can …

Content truncated to 500 words