Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure

Proceedings of the 28th International Conference on Computational Linguistics, pages 2642–2652 Barcelona, Spain (Online), December 8-13, 2020 2642 Molweni: A Challenge Multiparty Dialogue-based Machine Reading Comprehension Dataset with Discourse Structure Jiaqi Li1, Ming Liu1,3, Min-Yen Kan2, Zihao Zheng1, Zekun Wang1 Wenqiang Lei2, Ting Liu1,3, Bing Qin1,3∗ 1. Harbin Institute of Technology, Harbin, China. 2. National University of Singapore, Singapore. 3. Peng Cheng Laboratory, Shenzhen, China. {jqli, mliu, zhzheng,zkwang,tliu,qinb}@ir.hit.edu.cn kanmy@comp.nus.edu.sg, wenqianglei@gmail.com Abstract Research into the area of multiparty dialog has grown considerably over recent years. We present the Molweni dataset1, a machine reading comprehension (MRC) dataset with discourse structure built over multiparty dialog. Molweni’s source samples from the Ubuntu Chat Corpus, including 10,000 dialogs comprising 88,303 utterances. We annotate 30,066 questions on this corpus, including both answerable and unanswerable questions. Molweni also uniquely contributes discourse dependency annotations in a modified Segmented Discourse Representation Theory (SDRT; (Asher et al., 2016)) style for all of its multiparty dialogs, contributing large-scale (78,245 annotated discourse relations) data to bear on the task of multiparty dialog discourse parsing. Our experiments show that Molweni is a challenging dataset for current MRC models: BERT-wwm, a current, strong SQuAD 2.0 performer, achieves only 67.7% F1 on Molweni’s questions, a 20+% significant drop as compared against its SQuAD 2.0 performance. 1 Introduction Research into multiparty dialog has recently grown considerably, partially due to the growing ubiquity of dialog agents. Multiparty dialog applications such as discourse parsing and meeting summarization are now mainstream research (Shi and Huang, 2019; Hu et al., 2019; Li et al., 2019; Zhao et al., 2019; Sun et al., 2019; Perret et al., 2016; Afantenos et al., 2015). Such applications must consider the more complex, graphical nature of discourse structure: coherence between adjacent utterances is not a given, unlike standard prose where sequential guarantees hold. In a separate vein, the area of machine reading comprehension (MRC) research has also made unbridled progress recently. Most existing datasets for machine reading comprehension (MRC) adopt well-written prose passages and historical questions as inputs (Richardson et al., 2013; Rajpurkar et al., 2016; Lai et al., 2017; Choi et al., 2018; Reddy et al., 2019). Reading comprehension for dialog — as the intersection of these two areas — has naturally begun to attract interest. Ma et al. (2018) constructed a small dataset for passage completion on multiparty dialog, but which has been easily dispatched by CNN+LSTM models using attention. The DREAM corpus (Sun et al., 2019) is an MRC dataset for dialog, but only features a minute fraction (1%) of multiparty dialog. FriendsQA is a small-scale span-based MRC dataset for multiparty dialog, which derives from TV show Friends, including 1,222 dialogs and 10,610 questions (Yang and Choi, 2019). The limited number of dialogs in FriendsQA makes it infeasible to train more complex model to represent multiparty dialogs due to overfitting, and the lack of annotated discourse structure prevents models from making full use of the characteristics of multiparty dialog. Dialog-based MRC thus varies from other MRC variants in two key aspects: C1. …

Content truncated to 500 words