Dataset Creation for Ranking Constructive News Comments

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2619–2626 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2619 Dataset Creation for Ranking Constructive News Comments Soichiro Fujita,† Hayato Kobayashi,‡ and Manabu Okumura† † Tokyo Institute of Technology ‡ Yahoo Japan Corporation / RIKEN AIP {fujiso@lr.,oku@}pi.titech.ac.jp, hakobaya@yahoo-corp.jp Abstract Ranking comments on an online news ser- vice is a practically important task for the ser- vice provider, and thus there have been many studies on this task. However, most of them considered users’ positive feedback, such as “Like”-button clicks, as a quality measure. In this paper, we address directly evaluating the quality of comments on the basis of “construc- tiveness,” separately from user feedback. To this end, we create a new dataset including 100K+ Japanese comments with constructive- ness scores (C-scores). Our experiments clar- ify (a) C-scores are not always related to users’ positive feedback and (b) the performance of pairwise ranking models tends to be more en- hanced by the variation in comments than that in articles. 1 Introduction Users’ comments on an online news service can be regarded as beneficial content (often called user- generated content1) for service providers because users can obtain supplementary information about news articles through other users’ opinions. Given that comment visibility is a part of the user ex- perience, ranking comments is practically impor- tant. For example, Figure 1 shows a page dis- playing comments on a Japanese news portal, Ya- hoo! News.2 The page has a list of comments (displayed below articles), and each comment has buttons for user feedback (“Like,” “Dislike,” and “Reply”). There have been many comment ranking stud- ies (Hsu et al., 2009; Das Sarma et al., 2010; Brand and Van Der Merwe, 2014; Wei et al., 2016) with users’ positive feedback for a comment (e.g., “Like”- or “Upvote”-button clicks) serving as the 1https://en.wikipedia.org/wiki/ User-generated_content 2https://news.yahoo.co.jp/ Figure 1: Examples of comments on Yahoo! News. quality measure. However, this type of measure- ment has two drawbacks: (a) user feedback does not always satisfy the service provider’s needs, such as to create a fair place, and (b) user feedback will be biased by where comments appear in a comment thread. A typical situation for (a) can be seen in political comments, where the “goodness” of the comment will be decided on the basis of the political views of the majority of the users rather than its quality. The situation for (b) can be illus- trated by a case where earlier comments tend to re- ceive more feedback since they will be displayed at the top of the page, which implies later com- ments will be ignored irrespective of their quality. In this paper, we directly evaluate the qual- ity of comments separately from user feedback, focusing on their “constructiveness,” as studied in (Napoles et al., 2017; Kolhatkar and Taboada, 2017). This quality measure is reasonable for services in that displaying constructive comments can stimulate discussion on a news article, which makes the user-generated …

Content truncated to 500 words