Efficient Near-Duplicate Detection for Q&A Forum

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1001–1009, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Efficient Near-Duplicate Detection for Q&A Forum Yan Wu, Qi Zhang, Xuanjing Huang Fudan University School of Computer Science {10210240075,qz,xjhuang}@fudan.edu.cn Abstract This paper addresses the issue of re- dundant data in large-scale collections of Q&A forums. We propose and evaluate a novel algorithm for auto- matically detecting the near-duplicate Q&A threads. The main idea is to use the distributed index and Map- Reduce framework to calculate pair- wise similarity and identify redundant data fast and scalably. The proposed method was evaluated on a real-world data collection crawled from a popu- lar Q&A forum. Experimental results show that our proposed method can effectively and efficiently detect near- duplicate content in large web collec- tions. 1 INTRODUCTION There is a rise in popularity of Question and Answering forums in recent years. The forums allow users to post, browse, search and an- swer questions. Q&A forum acts not only as a medium for knowledge sharing , but also as a place in which one can seek advice, and sat- isfy others’ curiosity about a countless number of things (Adamic et al., 2008). However, be- cause of the ever-increasing growth of it, and the fact that users are not always experts in the areas they post threads on, duplicate con- tent becomes a serious issue. And, most of the current Q&A forums haven’t had a effi- cient mechanism to identify threads with near- duplicate content. As a consequence, users have to go through different versions of du- plicate or near-duplicate content, and are of- ten frustrated by it. Baidu Zhidao1 is one of the largest Q&A forums in China. It contains 1http://zhidao.baidu.com more than 100 million question and answer pairs. Because of the increasing popularity and the number of users, it also encounters the long-standing problem – content duplication. For example, there are more than four hundred question-answer pairs which contain the same content “When is the birthday of Jay?” Other Q&A forums are facing the similar problem. Along with the increasing requirements and the limitations of manual methods, there have been growing research activities in duplicate detection, during the past few years. Com- mon automatical methods to detect dupli- cates are copy detection or near-duplicate de- tection (Gionis et al., 1999; Muthmann et al., 2009; Shivakumar and Garcia-Molina, 1995; Shivakumar and Garcia-Molina, 1999; Theobald et al., 2008; Zhang et al., 2010). Most of the current near-duplication detec- tion approaches usually focus on the document level to figure out the web pages with different framing, navigation bar, and advertisements, but duplicate content. However, unlike dupli- cated web-pages, forum threads have the fol- lowing differences: 1. Forum threads contain additional struc- tured meta information, e.g. title, tags, external/internal links, etc. So, the method used to detect near-duplicate of the forum thread is to the web-page. 2. The average length of threads is usually less than the news articles. Adamic et al. reported that …

Content truncated to 500 words