Crowdsourcing Interaction Logs to Understand Text Reuse from the Web

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1212–1221, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Crowdsourcing Interaction Logs to Understand Text Reuse from the Web Martin Potthast Matthias Hagen Michael Völske Benno Stein Bauhaus-Universität Weimar 99421 Weimar, Germany <first name>.<last name>@uni-weimar.de Abstract We report on the construction of the Webis text reuse corpus 2012 for advanced re- search on text reuse. The corpus compiles manually written documents obtained from a completely controlled, yet representative environment that emulates the web. Each of the 297 documents in the corpus is about one of the 150 topics used at the TREC Web Tracks 2009–2011, thus forming a strong connection with existing evaluation efforts. Writers, hired at the crowdsourc- ing platform oDesk, had to retrieve sources for a given topic and to reuse text from what they found. Part of the corpus are detailed interaction logs that consistently cover the search for sources as well as the creation of documents. This will allow for in-depth analyses of how text is composed if a writer is at liberty to reuse texts from a third party—a setting which has not been studied so far. In addition, the corpus pro- vides an original resource for the evalua- tion of text reuse and plagiarism detectors, where currently only less realistic resources are employed. 1 Introduction The web has become one of the most common sources for text reuse. When reusing text from the web, humans may follow a three step ap- proach shown in Figure 1: searching for appro- priate sources on a given topic, copying of text from selected sources, modification and paraphras- ing of the copied text. A considerable body of research deals with the detection of text reuse, and, in particular, with the detection of cases of plagia- rism (i.e., the reuse of text with the intent of disguis- ing the fact that text has been reused). Similarly, a large number of commercial software systems is being developed whose purpose is the detection of plagiarism. Both the developers of these systems as well as researchers working on the subject matter frequently claim their approaches to be searching the entire web or, at least, to be scalable to web size. However, there is hardly any evidence to substantiate this claim—rather the opposite can be observed: commercial plagiarism detectors have not been found to reliably identify plagiarism from the web (Köhler and Weber-Wulff, 2010), and the evaluation of research prototypes even under lab- oratory conditions shows that there is still a long way to go (Potthast et al., 2010b). We explain the disappointing state of the art by the lack of realistic, large-scale evaluation resources. With our work, we want to contribute to closing the gap. In this regard the paper in hand introduces the Webis text reuse corpus 2012 (Webis-TRC-12), which, for the first time, emulates the entire process of reusing text from the web, both at scale and in a controlled environment. The corpus comprises a …

Content truncated to 500 words