Genre-Oriented Web Content Extraction with Deep Convolutional Neural Networks and Statistical Methods

Genre-Oriented Web Content Extraction with Deep Convolutional Neural Networks and Statistical Methods Bao-Dai Nguyen-Hoang Knorex Vietnam Co., Ltd. 46 Bach Dang, Tan Binh, Ho Chi Minh city, Vietnam dai_nguyen@knorex.com Bao-Tran Pham-Hong Knorex Vietnam Co., Ltd 46 Bach Dang, Tan Binh Ho Chi Minh city, Vietnam tran_pham@knorex.com Yiping Jin Knorex Pte. Ltd. 2 Science Park Drive, Singapore 118222 jinyiping@knorex.com Phu T. V. Le Knorex Pte. Ltd. 2 Science Park Drive, Singapore 118222 le.phu@knorex.com Abstract Extracting clean textual content from the Web is the first and an essential step to resolve most of down-stream natural language processing tasks. Previous works in web content extrac- tion focus mainly on web pages with a single main block of textual content, such as news ar- ticles and blog posts. They employ techniques that rely largely on the HTML structure of the page to extract the main content. Little attention has been paid to recogniz- ing different genres of web pages, which can have a tremendous impact on the accu- racy of the content extraction algorithms. In this paper, we propose Genre-Oriented Con- tent Extraction (GOCE), a novel content ex- traction framework consisting of two steps: web genre classification and content extrac- tion algorithms based on the detected genre. GOCE first employs state-of-the-art convolu- tional neural network models to classify the genre of the web page given a rendered image of the page. Then it applies content extrac- tion algorithm utilizing the genre information for a more robust and accurate textual con- tent extraction. Experiments show that GOCE achieves promising results in both web genre classification and the end-to-end content ex- traction task. Furthermore, GOCE greatly im- proves from a well-cited competitive baseline. We will also publish our dataset which con- tains annotations of both the web genre and the gold-standard textual content. 1 Introduction Web content extraction is the task of pulling the main textual content out of web pages while remov- ing noise such as clutters (for e.g. HTML tags, scripts, etc.) and boilerplate contents (for e.g. nav- igation bar, headers, footers, etc.) (Gupta et al., 2003). Humans can easily identify the main content based on the layout and visuals of web pages. How- ever, it is not trivial for a computer program to ac- curately detect the meaningful content and skip the noise due to the complex and dynamic content lay- out of web pages. Content extraction algorithms promise multiple applications by extracting only the important and relevant content from web pages. Firstly, the ex- tracted content is more readable for humans, which is an important criteria for third-party data con- sumers, such as an application that subscribes to the content of a website. Secondly, by removing the noisy textual content such as advertisements, headers and footers from the web page, it helps to improve the performance of down-stream natu- ral language processing tasks, such as text classifi- cation (Jena and Kamila, 2013) and text summarisa- tion (Joseph and Jeba, 2016). Kohlschütter et al. (2010) introduced a content extraction algorithm that uses heuristic …

Content truncated to 500 words