Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 649–662, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 649 Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations Mingda Chen2∗ Zewei Chu1∗ Kevin Gimpel2 1University of Chicago, IL, USA 2Toyota Technological Institute at Chicago, IL, USA {mchen,kgimpel}@ttic.edu,zeweichu@uchicao.edu Abstract Prior work on pretrained sentence embeddings and benchmarks focuses on the capabilities of representations for stand-alone sentences. We propose DiscoEval, a test suite of tasks to eval- uate whether sentence representations include information about the role of a sentence in its discourse context. We also propose a variety of training objectives that make use of natural annotations from Wikipedia to build sentence encoders capable of modeling discourse in- formation. We benchmark sentence encoders trained with our proposed objectives, as well as other popular pretrained sentence encoders, on DiscoEval and other sentence evaluation tasks. Empirically, we show that these train- ing objectives help to encode different aspects of information from the surrounding document structure. Moreover, BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018a) demon- strate strong performance across DiscoEval tasks with individual hidden layers showing different characteristics.1 1 Introduction Pretrained sentence representations have been found useful in various downstream tasks such as visual question answering (Tapaswi et al., 2016), script inference (Pichotta and Mooney, 2016), and information retrieval (Le and Mikolov, 2014; Palangi et al., 2016). Benchmark datasets (Adi et al., 2017; Conneau and Kiela, 2018; Wang et al., 2018a, 2019) have been proposed to evaluate the encoded knowledge, where the focus has been pri- marily on natural language understanding capabil- ities of the representation of a stand-alone sen- tence, such as its semantic roles, rather than the broader context in which it is situated. ∗Equal contribution. Listed in alphabetical order. 1Data processing and evaluation scripts are available at https://github.com/ZeweiChu/DiscoEval. NN-Comparison NS-Attribution 1 2 3 [The European Community’s consumer price index rose a provisional 0.6% in September from August]1 [and was up 5.3% from September 1988,]2 [according to Eurostat, the EC’s statistical agency.]3 Figure 1: An RST discourse tree from the RST Dis- course Treebank. “N” represents “nucleus”, contain- ing basic information for the relation. “S” represents “satellite”, containing additional information about the nucleus. In this paper, we seek to incorporate and eval- uate discourse knowledge in general purpose sen- tence representations. A discourse is a coherent, structured group of sentences that acts as a funda- mental type of structure in natural language (Ju- rafsky and Martin, 2009). A discourse structure is often characterized by the arrangement of se- mantic elements across multiple sentences, such as entities and pronouns. The simplest such ar- rangement (i.e., linearly-structured) can be under- stood as sentence ordering, where the structure is manifested in the timing of introducing entities. Deeper discourse structures use more complex re- lations among sentences (e.g., tree-structured; see Figure 1). Theoretically, discourse structures have been approached through Centering Theory (Grosz et al., 1995) for …

Content truncated to 500 words