Finding Convincing Arguments Using Scalable Bayesian Preference Learning

Finding Convincing Arguments Using Scalable Bayesian Preference Learning Edwin Simpson and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science Technische Universit¨at Darmstadt {simpson, gurevych}@ukp.informatik.tu-darmstadt.de Abstract We introduce a scalable Bayesian preference learning method for identifying convincing ar- guments in the absence of gold-standard rat- ings or rankings. In contrast to previous work, we avoid the need for separate methods to perform quality control on training data, pre- dict rankings and perform pairwise classifica- tion. Bayesian approaches are an effective so- lution when faced with sparse or noisy train- ing data, but have not previously been used to identify convincing arguments. One issue is scalability, which we address by develop- ing a stochastic variational inference method for Gaussian process (GP) preference learn- ing. We show how our method can be ap- plied to predict argument convincingness from crowdsourced data, outperforming the previ- ous state-of-the-art, particularly when trained with small amounts of unreliable data. We demonstrate how the Bayesian approach en- ables more effective active learning, thereby reducing the amount of data required to iden- tify convincing arguments for new users and domains. While word embeddings are princi- pally used with neural networks, our results show that word embeddings in combination with linguistic features also benefit GPs when predicting argument convincingness. 1 Introduction Arguments are intended to persuade the audience of a particular point of view and are an important way for humans to reason about controversial top- ics (Mercier and Sperber, 2011). The amount of ar- gumentative text on any chosen subject can, how- Topic: “William Farquhar ought to be honoured as the rightful founder of Singapore”. Stance: “No, it is Raffles!” Argument 1: HE HAS A BOSS(RAFFLES) HE HAS TO FOLLOW HIM AND NOT GO ABOUT DOING ANYTHING ELSE... Argument 2: Raffles conceived a town plan to re- model Singapore into a modern city. The plan con- sisted of separate areas for different... Crowdsourced labels: {2 ≻ 1, 1 ≻ 2, 2 ≻ 1} Figure 1: Example argument pair from an online debate. ever, overwhelm a reader. Consider the scale of his- torical text archives or social media platforms with millions of users. Automated methods could help readers overcome this challenge by identifying high- quality, persuasive arguments from both sides of a debate. Theoretical approaches for assessing argument quality have proved difficult to apply to everyday arguments (Boudry et al., 2015). Empirical ma- chine learning approaches instead train models us- ing example judgments of arguments, such as those shown in Figure 1. Previous approaches to obtain- ing such judgments include training annotators to assign scores from 1-6 (Persing and Ng, 2017), ask- ing annotators for simple binary or three-class cat- egories (Wei et al., 2016b), and aggregating binary votes from multiple people (Wei et al., 2016a; Tan et al., 2016). However, these approaches are limited by the cost of training annotators, a highly restricted set of categories, or the need for multiple annotators per document. 357 Transactions of the Association for Computational Linguistics, vol. 6, pp. 357–371, …

Content truncated to 500 words