Efficient Pairwise Annotation of Argument Quality

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5772–5781 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5772 Efficient Pairwise Annotation of Argument Quality Lukas Gienapp1 Benno Stein2 Matthias Hagen3 Martin Potthast1 1 Leipzig University 2 Bauhaus-Universität Weimar 3 Martin-Luther-Universität Halle-Wittenberg Abstract We present an efficient annotation framework for argument quality, a feature difficult to be measured reliably as per previous work. stochastic transitivity model is combined withA an effective sampling strategy to infer high- quality labels with low effort from crowd- sourced pairwise judgments. The model’s ca- pabilities are showcased by compiling Webis- ArgQuality-20, an argument quality corpus that comprises scores for rhetorical, logical, dialectical, and overall quality inferred from a total of 41,859 pairwise judgments among 1,271 arguments. With up to 93% cost sav- ings, our approach significantly outperforms existing annotation procedures. Furthermore, novel insight into argument quality is provided through statistical analysis, and a new aggrega- tion method to infer overall quality from indi- vidual quality dimensions is proposed. 1 Introduction For a broad variety of tasks, such as argument min- ing, argument retrieval, argumentation generation, and question answering, compiling labeled data for argument quality remains an important prerequi- site, yet, also a difficult problem. Most commonly, human assessors have been presented with one ar- gument at a time and then asked to assign labels on a graded quality scale ⟨0, 1, 2⟩ with label de- scriptions such as (0) “low quality”, (1) “medium quality” and (2) “high quality” for guidance. In previous work, this was usually done concur- rently for multiple orthogonal sub-dimensions of argument quality; judging the overall quality of an argument has been deemed complex (Wachsmuth et al., 2017). But on closer inspection, even the more specialized quality dimensions considered are difficult to be assessed as evidenced by the low reliability scores reported. Especially crowdsourc- ing suffers from assessors often having different reference frames to base their judgments on and task instructions being nondescript and therefore unhelpful in ensuring consistency. Employing ex- perts, however, not only comes at a significantly higher cost per label; despite their expertise, even experts did not achieve more reliable judgments. We pursue an alternative approach: stochastic transitivity modeling based on pairwise judgments of arguments. This enables the employment of lay- men; the decisions required from them remain com- parably simple and expect neither prior knowledge nor a common reference frame, while the labels that can be derived from their judgments still ex- hibit a high accuracy and informativeness. Though pairwise judgment has already been considered for assessment of argument quality (Habernal and Gurevych, 2016; Toledo et al., 2019), its significant cost overhead has hindered widespread application. We explore the lower bound of effort needed to infer labels of sufficient quality. We combine a pairwise model with a highly effective offline sampling strategy to minimize the set of needed pairwise judgments, saving up to 93% of the ef- fort of an exhaustive comparison. As part of this work, we release the Webis Argument …

Content truncated to 500 words