Automatic Wrapper Generation and Maintenance

Automatic Wrapper Generation and Maintenance Yingju Xia, Yuhang Yang, Fujiang Ge, Shu Zhang, Hao Yu Fujitsu Research & Development Center Co.,LTD. 15F Tower A, Ocean International Center, No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025 yjxia@cn.fujitsu.com Abstract. This paper investigates automatic wrapper generation and maintenance for Forums, Blogs and News web sites. Web pages are increasingly dynamically generated using a common template populated with data from databases. This paper proposes a novel method that uses tree alignment and transfer learning method to generate the wrapper from this kind of web pages. The tree alignment algorithm is adopted to find the best matching structure of the input web pages. A kind of linear regression method is employed to get the weight of different tag-matching. A transfer learning method is adopted to find the most likely content block. A wrapper built on the most probable content block and the repeating patterns extracts data from web pages. The wrapper maintenance arises because web source may experiment changes that invalidate the current wrappers. This paper presents a wrapper maintenance method using a log likelihood ratio test for detecting the change points on the similarity series which gotten from the wrapper and input web pages. The wrapper generation method is applied to generate a wrapper once the web source change is detected. Experimental results show that the method achieves high accuracy and has steady performance Keywords: Data extraction, Wrapper generation, Wrapper maintenance. 1 Introduction Web-based information is typically formatted to be read by human users, not by computer applications. Information agents are being proposed which automatically extract information from multiple websites. Data is typically extracted from web sources by writing specialized programs, called wrappers (Laender et al. 2002), which identify data of interest and map them to a suitable format. Many approaches have been reported in the literature for wrapper generation. Detailed discussions of various approaches can be found in several surveys (Chang et al., 2006; Laender et al., 2002). Early approaches were based on manual techniques (Atzeni and Mecca, 1997; Crescenzi and Mecca, 1998; Huck et al., 1998; Sahuguet and Azavant, 1999). By observing a web page and its source code, the programmer find some patterns from the page and then write a program to extract data from the web pages. A key problem with manually coded wrappers is that writing them is a difficult and labor-intensive task, and tends to be brittle and difficult to maintain. Other approaches have some degrees of automation. In semi-automatic approaches (Cohen et al., 2002; Irmak and Suel, 2006; Kushmerick, 2000; Muslea et al., 1999; Pinto et al., 2003; Wang and Hu, 2002; Zheng et al., 2007), a set of extraction rules are learnt from a set of manually labeled pages or data records. These rules are then used to extract data items from similar pages. This method still requires substantial manual efforts. In automatic methods, Arasu and Hector (2003), Chang and Liu (2001) and Crescenzi et al. (2001) found patterns or grammars from multiple …

Content truncated to 500 words