The Role of Context and Uncertainty in Shallow Discourse Parsing

Proceedings of the 29th International Conference on Computational Linguistics, pages 797–811 October 12–17, 2022. 797 The Role of Context and Uncertainty in Shallow Discourse Parsing Katherine Atwell†, Remi Choi†, Junyi Jessy Li§, Malihe Alikhani† †Department of Computer Science, University of Pittsburgh §Department of Linguistics, The University of Texas at Austin {kaa139, theresachoi, malihe}@pitt.edu, jessy@utexas.edu Abstract Discourse parsing has proven to be useful for a number of NLP tasks that require complex reasoning. However, over a decade since the advent of the Penn Discourse Treebank, predict- ing implicit discourse relations in text remains challenging. There are several possible rea- sons for this, and we hypothesize that models should be exposed to more context as it plays an important role in accurate human annota- tion; meanwhile adding uncertainty measures can improve model accuracy and calibration. To thoroughly investigate this phenomenon, we perform a series of experiments to determine 1) the effects of context on human judgments, and 2) the effect of quantifying uncertainty with an- notator confidence ratings on model accuracy and calibration (which we measure using the Brier score (Brier et al., 1950)). We find that including annotator accuracy and confidence improves model accuracy, and incorporating confidence in the model’s temperature function can lead to models with significantly better- calibrated confidence measures. We also find some insightful qualitative results regarding hu- man and model behavior on these datasets. 1 Introduction The context of an utterance influences the interpre- tation. Linguistics, philosophy, cognitive science and neuroscience (Lewis, 1980; Glanzberg, 2002; Thompson-Schill, 2003) studies have shown the effect of context in text interpretation. In this pa- per, we hypothesize that context can influence the prediction of discourse relation labels. Motivating the need to discover the effects of context on an- notation accuracy is the following example, where argument 1 is bolded and argument 2 is italicized: (1) the fund’s 25% leverage has jacked up its interest income As long as I am borrowing at 9.9% and each {bond} yields over that, it enhances the yield Context But when the market moves against the fund, investors lose more than other junk holders … he maintains. Arg 1 Arg 2 the fund’s 25% leverage has jacked up its interest income As long as I am borrowing at 9.9% … it enhances the yield Human confidence Predicted confidence } + Predicted relation Contingency.Cause Figure 1: A diagram of our model, in which we use contextual information to predict uncertainty and pass the results into a model that predicts the discourse rela- tion. We experiment with several different architectures and machine learning methods for utilizing uncertainty, including concatenation. The lack of contextual information makes it diffi- cult to determine the proper discourse relation here, and our annotator chose Expansion.Conjunction as a result. However, when more context is added, the passage presented to the annotator is as follows: (2) But when the market moves against the fund, investors lose more than other junk holders because the market decline is mag- nified by the amount the fund is leveraged. …

Content truncated to 500 words