Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations

Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations Yudai Kishimoto, Shinnosuke Sawada, Yugo Murawaki, Daisuke Kawahara, Sadao Kurohashi Graduate School of Informatics, Kyoto University Kyoto, Japan {kishimoto, sawada, murawaki, dk, kuro}@nlp.ist.i.kyoto-u.ac.jp Abstract Although discourse parsing is an important and fundamental task in natural language processing, few languages have corpora anno- tated with discourse relations and if any, they are small in size. Creating a new corpus of discourse relations by hand is costly and time-consuming. To cope with this problem, Kawahara et al. (2014) constructed a Japanese corpus with discourse annotations through crowdsourcing. However, they did not evaluate the quality of the annotation. In this paper, we evaluate the quality of the annotation using expert annotations. We find out that crowdsourcing-based annotation still leaves much room for improvement. Based on the error analysis, we propose improvement techniques based on language tests. We re-annotated the corpus with discourse annotations using the improvement techniques, and achieved approximately 3% improvement in F-measure. We will make re-annotated data publicly avail- able. Keywords: discourse annotation, crowdsourcing 1. Introduction Humans understand text not by interpreting clauses or sen- tences individually, but by linking such a text fragment with another in a particular context. To allow computers to un- derstand text, it is essential to capture the precise relations between these text fragments. The task of analyzing these relations is called discourse parsing. Discourse relations are conventionally divided into two types: explicit and im- plicit. Explicit relations are overtly marked with discourse connectives such as “and” and “however.” By contrast, im- plicit relations lack discourse connectives. Discourse parsing is an important and fundamental task in natural language processing. Systems for discourse parsing are, however, available only for major languages, such as English and Chinese. This is because few languages have corpora annotated with discourse relations and if any, they are small in size. Moreover, creating a new corpus of dis- course relations by hand is costly and time-consuming. Kawahara et al. (2014) addressed this problem by using crowdsourcing. Crowdsourcing allows for cheap and speedy annotation. However, crowdsourcing-based anno- tation tends to be of poorer quality than expert annotation. In this paper, we evaluate the quality of the annotations pro- duced by Kawahara et al. (2014). By asking experts to an- notate a part of the corpus, we evaluate the quality of the annotation. Next, we analyze the corpus and find two prob- lems. Then we propose solutions to each of these problems. Finally, we re-annotate the corpus with discourse relations through crowdsourcing. Experimental results show that the accuracy is improved by an F-measure of 3%. 2. Related Work There are several corpora with discourse annotations for English, such as the Penn Discourse Treebank (PDTB) (Prasad et al., 2008) and the RST Discourse Treebank (RST-DT) (Carlson et al., 2001). PDTB’s and RST-DT’s annotations are done as another layer on the Wall Street Journal section of the Penn Treebank. The PDTB consists 2,159 articles and each discourse relation con- sists of two text spans (arguments) and a relation la- bel. …

Content truncated to 500 words