Training Data Enrichment for Infrequent Discourse Relations

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2603–2614, Osaka, Japan, December 11-17 2016. Training Data Enrichment for Infrequent Discourse Relations Kailang Jiang, Giuseppe Carenini, Raymond T. Ng Department of Computer Science University of British Columbia Vancouver, BC, Canada, V6T 1Z4 {jiangkl, carenini, rng}@cs.ubc.ca Abstract Discourse parsing is a popular technique widely used in text understanding, sentiment analysis and other NLP tasks. However, for most discourse parsers, the performance varies significantly across different discourse relations. In this paper, we first validate the underfitting hypothesis, i.e., the less frequent a relation is in the training data, the poorer the performance on that relation. We then explore how to increase the number of positive training instances, without resorting to manually creating additional labeled data. We propose a training data enrichment framework that relies on co-training of two different discourse parsers on unlabeled documents. Importantly, we show that co-training alone is not sufficient. The framework requires a filtering step to ensure that only “good quality” unlabeled documents can be used for enrichment and re-training. We propose and evaluate two ways to perform the filtering. The first is to use an agreement score between the two parsers. The second is to use only the confidence score of the faster parser. Our empirical results show that agreement score can help to boost the performance on infrequent relations, and that the confidence score is a viable approximation of the agreement score for infrequent relations. 1 Introduction Discourse parsing is widely used in text understanding (Allen et al., 2014), sentiment analysis (Bha- tia et al., 2015) and other NLP tasks (Guzm´an et al., 2014) (Gerani et al., 2016). A multi-sentential discourse parser takes a document as input, and returns its discourse structure that shows how clauses and sentences are related in the document, via the use of various discourse relations. For instance, the benchmark RST-DT dataset (Carlson et al., 2001) uses 18 discourse relations. Studies in the past decade on discourse parsing, such as (Ji and Eisenstein, 2014), (Feng and Hirst, 2014), and (Joty et al., 2015), greatly improved the performance of discourse parsing in general. However, it has been observed that the performance across the discourse relations varies significantly (Joty et al., 2015), and that poor per- formance may be linked to underfitting, i.e., a lack of training data (Feng and Hirst, 2014). In this paper, we investigate the underfitting hypothesis and study how to improve the situation. Dif- ferent discourse relations are usually unevenly distributed in a dataset, and some of them occur much less frequently than other relations. We call the former the infrequent relations. For example, in the very popular corpus — Rhetorical Structure Theory Discourse Treebank (RST-DT) (Carlson et al., 2001) which contains 385 documents, the frequency of “Elaboration” is 31.04%, while the frequency of “Sum- mary” is only 0.88%. In another benchmark corpus the Penn Discourse Treebank (PDTB) 2.0 (Prasad et al., 2008), which contains about 2400 documents with discourse relations labeled for each pair …

Content truncated to 500 words