Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5709–5714 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5709 Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering Changmao Li Department of Computer Science Emory University Atlanta, GA, USA changmao.li@emory.edu Jinho D. Choi Department of Computer Science Emory University Atlanta, GA, USA jinho.choi@emory.edu Abstract We introduce a novel approach to transformers that learns hierarchical representations in mul- tiparty dialogue. First, three language model- ing tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better un- derstanding in dialogue contexts. Then, multi- task learning between the utterance prediction and the token span prediction is applied to fine- tune for span-based question answering (QA). Our approach is evaluated on the FRIENDSQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively. 1 Introduction Transformer-based contextualized embedding ap- proaches such as BERT (Devlin et al., 2019), XLM (CONNEAU and Lample, 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), and AlBERT (Lan et al., 2019) have re-established the state-of-the-art for practically all question answer- ing (QA) tasks on not only general domain datasets such as SQUAD (Rajpurkar et al., 2016, 2018), MS MARCO (Nguyen et al., 2016), TRIVIAQA (Joshi et al., 2017), NEWSQA (Trischler et al., 2017), or NARRATIVEQA (Koisk et al., 2018), but also multi- turn question datasets such as SQA (Iyyer et al., 2017), QUAC (Choi et al., 2018), COQA (Reddy et al., 2019), or CQA (Talmor and Berant, 2018). However, for span-based QA where the evidence documents are in the form of multiparty dialogue, the performance is still poor even with the latest transformer models (Sun et al., 2019; Yang and Choi, 2019) due to the challenges in representing utterances composed by heterogeneous speakers. Several limitations can be expected for language models trained on general domains to process dia- logue. First, most of these models are pre-trained on formal writing, which is notably different from colloquial writing in dialogue; thus, fine-tuning for the end tasks is often not sufficient enough to build robust dialogue models. Second, unlike sentences in a wiki or news article written by one author with a coherent topic, utterances in a dialogue are from multiple speakers who may talk about different top- ics in distinct manners such that they should not be represented by simply concatenating, but rather as sub-documents interconnected to one another. This paper presents a novel approach to the latest transformers that learns hierarchical embeddings for tokens and utterances for a better understand- ing in dialogue contexts. While fine-tuning for span-based QA, every utterance as well as the ques- tion are separated encoded and multi-head atten- tions and additional transformers are built on the token and utterance embeddings respectively to pro- vide a more comprehensive view of the dialogue to the QA model. As a result, our model achieves …

Content truncated to 500 words