BERT-enhanced Relational Sentence Ordering Network

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6310–6320, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6310 BERT-enhanced Relational Sentence Ordering Network Baiyun Cui1, Yingming Li1∗, and Zhongfei Zhang2 1College of Information Science and Electronic Engineering, Zhejiang University, China 2Computer Science Department, Binghamton University, Binghamton, NY, USA baiyunc@yahoo.com, yingming@zju.edu.cn, zzhang@binghamton.edu Abstract In this paper, we introduce a novel BERT- enhanced Relational Sentence Ordering Net- work (referred to as BERSON) by leverag- ing BERT for capturing a better dependency relationship among sentences to enhance the coherence modeling for the entire paragraph. In particular, we develop a new Relational Pointer Decoder (referred as RPD) by in- corporating the relative ordering information into the pointer network with a Deep Rela- tional Module (referred as DRM), which uti- lizes BERT to exploit the deep semantic con- nection and relative ordering between sen- tences. This enables us to strengthen both lo- cal and global dependencies among sentences. Extensive evaluations are conducted on six public datasets. The experimental results demonstrate the effectiveness and promise of BERSON, showing a significant improvement over the state-of-the-art by a wide margin. 1 Introduction Coherence modeling is one of the essential aspects of natural language processing (Xu et al., 2019; Mesgar et al., 2019; Moon et al., 2019; Farag and Yannakoudakis, 2019). A coherent text can fa- cilitate understanding and avoid the confusion for reading comprehension. The Sentence Ordering task (Barzilay and Lapata, 2008) aims to recon- struct a coherent paragraph from an unordered set of sentences and has shown to be beneficial to im- prove the coherence in many NLP tasks includ- ing multi-document summarization (Barzilay and Elhadad, 2002; Nallapati et al., 2017), conversa- tional analysis (Zeng et al., 2018), and text gener- ation (Konstas and Lapata, 2013; Holtzman et al., 2018). Table 1 shows an example of this task. In recent years, several approaches based on ranking or sorting frameworks have been devel- ∗Corresponding author An unordered set of sentences Coherent paragraph 1 Dan was walking during the night. 1 Dan was walking during the night. 3 They tried to steal his book bag. 2 A group of thieves surrounded him. 4 A bystander noticed them. 3 They tried to steal his book bag. 2 A group of thieves surrounded him. 4 A bystander noticed them. 5 But she continued to walk away. 5 But she continued to walk away. Table 1: Illustration of the sentence ordering task. It aims to reorganize an unordered set of sentences into a coherent paragraph. oped to deal with this task. RankTxNet (Ku- mar et al., 2020) computes a score for each sen- tence and sorts these scores with ranking based loss functions. Pairwise Model (Chen et al., 2016) adopts a pairwise ranking algorithm to learn the relative order of each sentence pair. B-TSort (Prabhumoye et al., 2020) predicts the constraint between two sentences and uses the topological sort technique to find the ordering. On the other hand, to better capture the global coherence, pointer network (Vinyals …

Content truncated to 500 words