Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog

Proceedings of NAACL-HLT 2018, pages 2039–2048 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog Kaixin Ma, Tomasz Jurczyk, Jinho D. Choi Math and Computer Science Emory University Atlanta, GA 30322, USA {kaixin.ma, tomasz.jurczyk, jinho.choi}@emory.edu Abstract This paper presents a new corpus and a robust deep learning architecture for a task in reading comprehension, passage completion, on multi- party dialog. Given a dialog in text and a pas- sage containing factual descriptions about the dialog where mentions of the characters are re- placed by blanks, the task is to fill the blanks with the most appropriate character names that reflect the contexts in the dialog. Since there is no dataset that challenges the task of passage completion in this genre, we create a corpus by selecting transcripts from a TV show that com- prise 1,681 dialogs, generating passages for each dialog through crowdsourcing, and anno- tating mentions of characters in both the dialog and the passages. Given this dataset, we build a deep neural model that integrates rich feature extraction from convolutional neural networks into sequence modeling in recurrent neural net- works, optimized by utterance and dialog level attentions. Our model outperforms the previ- ous state-of-the-art model on this task in a dif- ferent genre using bidirectional LSTM, show- ing a 13.0+% improvement for longer dialogs. Our analysis shows the effectiveness of the at- tention mechanisms and suggests a direction to machine comprehension on multiparty dialog. 1 Introduction Reading comprehension that challenges machine’s ability to understand a document through question answering has gained lots of interests. Most of the previous works for reading comprehension have fo- cused on either children’s stories (Richardson et al., 2013; Hill et al., 2016) or newswire (Hermann et al., 2015; Onishi et al., 2016). Few approaches have attempted comprehension on small talks, although they are evaluated on toy examples not suitable to project real-life performance (Weston et al., 2015). It is apparent that the main stream of reading com- prehension has not been on the genre of multiparty dialog although it is the most common and natural means of human communication. The volume of data accumulating from group chat or messaging continues to outpace data accumulation from other writing sources. The combination of available and rapidly developing analytic options, a marked need for dialogue processing, and the disproportionate generation of data from conversations through text platforms inspires us to create a corpus consisting of multiparty dialogs and develop learning models that make robust inference on their contexts. Passage completion is a popular method of evalu- ating reading comprehension that is adapted by sev- eral standardized tests (e.g., SAT, TOEFL, GRE). Given a document and a passage containing factual descriptions about the contexts in the document, the task replaces keywords in the passage with blanks and asks the reader to fill in the blanks. This task is particularly challenging when the document is in a form of dialog because it needs …

Content truncated to 500 words