Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1152–1158 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1152 Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives Yudai Kishimoto, Yugo Murawaki, Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan {kishimoto, murawaki, kuro} @nlp.ist.i.kyoto-u.ac.jp Abstract BERT, a neural network-based language model pre-trained on large corpora, is a breakthrough in natural language processing, signif- icantly outperforming previous state-of-the-art models in numerous tasks. However, there have been few reports on its application to implicit discourse relation classification, and it is not clear how BERT is best adapted to the task. In this paper, we test three methods of adaptation. (1) We perform additional pre-training on text tailored to discourse classification. (2) In expectation of knowledge transfer from explicit discourse relations to implicit discourse relations, we add a task named explicit connective prediction at the additional pre-training step. (3) To exploit implicit connectives given by treebank annotators, we add a task named implicit connective prediction at the fine-tuning step. We demonstrate that these three techniques can be combined straightforwardly in a single training pipeline. Through comprehensive experiments, we found that the first and second techniques provide additional gain while the last one did not. Keywords: discourse relation classification, BERT, explicit connective, implicit connective 1. Introduction Discourse relation classification, a task of recog- nizeing the semantic relations between two text spans, is beneficial for many NLP tasks includ- ing machine translation (Meyer et al., 2015), natu- ral language inference (Pan et al., 2018), summa- rization (Isonuma et al., 2019), and sentiment analy- sis (Saito et al., 2019). In the Penn Discourse TreeBank (PDTB) (Prasad et al., 2008), discourse relations are con- ventionally divided into two types: explicit and implicit. Explicit relations have strong cues named discourse con- nectives such as “because” and “however”, while implicit relations lack these cues. For this reason, the recognition of implicit relations is the bottleneck of discourse relation classification (Xue et al., 2016; Dai and Huang, 2019). In this paper, we investigate how to improve the per- formance of implicit discourse relation classification by applying BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2018). BERT is a Transformer-based neural network architecture with spe- cialized training procedures. It is pre-trained on large cor- pora like Wikipedia and BooksCorpus, and fine-tuned using task-specific datasets to transfer the pre-trained represen- tations to downstream tasks. Although BERT is concep- tually simple, it significantly outperforms previous state- of-the-art models in many natural language processing tasks such as reading comprehension (Devlin et al., 2018), syntactic analysis (Goldberg, 2019), and sentiment analy- sis (Xu et al., 2019). In implicit discourse relation classification, Nie et al. (2019) and Shi and Demberg (2019b) reported that BERT significantly outperformed previous state-of- the-art models. Considering the broader context of research in this area, however, we expect additional improvement to be achieved with task-specific adaptation. Table 1 summarizes recent studies and techniques …

Content truncated to 500 words