Sentiment-Stance-Specificity (SSS) Dataset: Identifying Support-based Entailment among Opinions.

Sentiment-Stance-Specificity (SSS) Dataset: Identifying Support-based Entailment among Opinions. Pavithra Rajendran, Danushka Bollegala, Simon Parsons University of Liverpool, University of Liverpool, Kings College London Pavithra.Rajendran@liverpool.ac.uk, danushka.bollegala@liverpool.ac.uk, simon.parsons@kcl.ac.uk Abstract Computational argumentation aims to model arguments as a set of premises that either support each other or collectively support a conclusion. We prepare three datasets of text-hypothesis pairs with support-based entailment based on opinions present in hotel reviews using a distant supervision approach. Support-based entailment is defined as the existence of a specific opinion (premise) that supports as well as entails a more general opinion and where these together support a generalised conclusion. A set of rules is proposed based on three different components — sentiment, stance and specificity to automatically predict support-based entailment. Two annotators manually annotated the relations among text-hypothesis pairs with an inter-rater agreement of 0.80. We compare the performance of the rules which gave an overall accuracy of 0.83. Further, we compare the performance of textual entailment under various conditions. The overall accuracy was 89.54%, 90.00% and 96.19% for our three datasets. Keywords: argument mining, stance classification, structured argumentation 1. Introduction Argument mining (Abbas and Sawamura, 2008; Palau and Moens, 2009) deals with the extraction of argument com- ponents and structures from natural language texts. has drawn attention from both the argumentation and NLPIt communities with the introduction of ArgMining work- shops 1. In computational argumentation, an argument can be defined as a collection of premises together (linked ar- gument) or individually (convergent argument) which are related to a conclusion (Palau and Moens, 2009). Each premise provides a support in the form of logical reason- ing for, or evidence in support of, the conclusion to which it is connected. It has been suggested that, in natural language texts, this support relation can be interpreted as meaning either (a) one premise is inferred from another premise (Janier et al., 2014) or (b) one premise provides evidence that supports another premise (Park and Cardie, 2014). In either case, it is natural to interpret the relationship as a form of entail- ment. In this paper, we consider a subtype of entailment, which we term support-based entailment, where a support rela- tion exists between the text and the hypothesis. Despite the unstructured nature of natural language texts, they pro- vide meta-linguistic attributes such as stance, sentiment, and specificity that can be exploited for detecting support- based entailment. We create a dataset of text-hypothesis pairs from opinions collected from a set of hotel reviews where the text pro- vides support to the corresponding hypothesis. As an exam- ple, we consider online reviews that are comprised largely of opinionated texts that talk about various aspects of a product or service. Consider the examples shown in Fig. 1 where we have two different reviews with overall star rat- ings of 1 and 2. There we see a collection of opinions, that is sentence-level statements that talk about one or more as- pects of a product or service. These are the basic units that 1https://argmining2017.wordpress.com/ we deal …

Content truncated to 500 words