RST Discourse Parsing with Second-Stage EDU-Level Pre-training

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4269 - 4280 May 22-27, 2022 c⃝2022 Association for Computational Linguistics RST Discourse Parsing with Second-Stage EDU-Level Pre-training Nan Yu1 and Meishan Zhang2 and Guohong Fu1,3∗and Min Zhang2 1School of Computer Science and Technology, Soochow University, China 2Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China 3Institute of Artificial Intelligence, Soochow University, China nyu@stu.suda.edu.cn, mason.zms@gmail.com, {ghfu,minzhang}@suda.edu.cn Abstract Pre-trained language models (PLMs) have shown great potentials in natural language pro- cessing (NLP) including rhetorical structure theory (RST) discourse parsing. Current PLMs are obtained by sentence-level pre-training, which is different from the basic processing unit, i.e. element discourse unit (EDU). To this end, we propose a second-stage EDU-level pre- training approach in this work, which presents two novel tasks to learn effective EDU represen- tations continually based on well pre-trained language models. Concretely, the two tasks are (1) next EDU prediction (NEP) and (2) dis- course marker prediction (DMP). We take a state-of-the-art transition-based neural parser as baseline, and adopt it with a light bi-gram EDU modification to effectively explore the EDU-level pre-trained EDU representation. Ex- perimental results on a benckmark dataset show that our method is highly effective, leading a 2.1-point improvement in F1-score. All codes and pre-trained models will be released pub- licly to facilitate future studies.1 1 Introduction Discourse analysis based on rhetorical structure theory (RST) has received increasing interest in the natural language processing (NLP) community (Yu et al., 2018; Liu et al., 2019a; Kobayashi et al., 2020; Zhang et al., 2020; Guz and Carenini, 2020; Koto et al., 2021; Zhang et al., 2021), which orga- nizes discourse output through a well-defined tree structure. Figure 1 shows an example of an RST constituent tree, where the leaf nodes are element discourse units (EDUs). Given an EDU sequence, RST discourse parsing aims to automatically con- struct a hierarchical constituent tree2. ∗Corresponding author. 1http://github.com/yunan4nlp/ E-NNRSTParser 2In this study, we focus on the tree construction task, as- suming the gold standard EDU as inputs. e1[CNW Corp. said] e2[the final step in the acquisition of the company has been completed with the merger of CNW with a subsidiary of Chicago & North Western Holdings Corp.] e3[As reported,] e4[CNW agreed to be acquired by a group of investors] e5[led by Blackstone Capital Partners Limited Partnership] e6[for $50 a share, or about $950 million.] e1 e2 e3 e4 e5 e6 elab-NS same-NN circ-SN attr-SN elab-SN Figure 1: An example of RST discourse tree. e1, e2, e3, e4, e5, and e6 are EDUs. elab, attr, circ and same are relations. NS, SN, and NN are nuclearities. The shift-reduce transition-based model has been widely adopted in RST discourse parsing (Yu et al., 2018; Mabona et al., 2019), building the con- stituent tree incrementally with multiple steps by a sequence of actions. These models take EDU- level features as inputs to score transition actions at each step. Recently, neural network models have achieved state-of-the-art performance for this task …

Content truncated to 500 words