JATE 2.0: Java Automatic Term Extraction with Apache Solr

JATE 2.0: Java Automatic Term Extraction with Apache Solr Ziqi Zhang, Jie Gao, Fabio Ciravegna Regent Court, 211 Portobello, Sheffield, UK, S1 4DP ziqi.zhang@sheffield.ac.uk, j.gao@sheffield.ac.uk, f.ciravegna@sheffield.ac.uk Abstract Automatic Term Extraction (ATE) or Recognition (ATR) is a fundamental processing step preceding many complex knowledge engineering tasks. However, few methods have been implemented as public tools and in particular, available as open-source freeware. Further, little effort is made to develop an adaptable and scalable framework that enables customization, development, and comparison of algorithms under a uniform environment. This paper introduces JATE 2.0, a complete remake of the free Java Automatic Term Extraction Toolkit (Zhang et al., 2008) delivering new features including: (1) highly modular, adaptable and scalable ATE thanks to integration with Apache Solr, the open source free-text indexing and search platform; (2) an extended collection of state-of-the-art algorithms. We carry out experiments on two well-known benchmarking datasets and compare the algorithms along the dimensions of effectiveness (precision) and efficiency (speed and memory consumption). To the best of our knowledge, this is by far the only free ATE library offering a flexible architecture and the most comprehensive collection of algorithms. Keywords: term extraction, term recognition, NLP, text mining, Solr, search, indexing 1. Introduction Automatic Term Extraction (or Recognition) is an impor- tant Natural Language Processing (NLP) task that deals with the extraction of terminologies from domain-specific textual corpora. ATE is widely used by both industries and researchers in many complex tasks, such as Informa- tion Retrieval (IR), machine translation, ontology engineer- ing and text summarization (Bowker, 2003; Brewster et al., 2007; Maynard et al., 2007). Over the years, there has been constant effort on researching new algorithms, adapting to different domains and/or languages, and creat- ing benchmarking datasets (Ananiadou, 1994; Church and Gale, 1995; Ahmad et al., 1999; Frantzi et al., 2000; Park et al., 2002; Matsuo and Ishizuka, 2003; Sclano and Ve- lardi, 2007; Rose et al., 2010; Spasi´c et al., 2013; Zadeh and Handschuh, 2014). Given the fundamental role of ATE in many tasks, a real benefit to the community would be releasing the tools to facilitate the development of downstream applications, also encouraging code-reuse, comparative studies, and fostering further research. Unfortunately, effort in this direction has been extremely limited. First, only very few tools have been published with readily available source code (Spasi´c et al., 2013) and some are proprietary (labs.translated.net, 2015; Yahoo!, 2015), or no longer maintained (Sclano and Ve- lardi, 2007)1. Second, different tools have been developed under different scenarios and evaluated in different domains using proprietary language resources, making it difficult for comparison. Third, it is unclear whether and how well these tools can adapt to different domain tasks and scale up to large data. To the best of our knowledge, the only effort towards ad- dressing these issues is the Java Automatic Term Extrac- tion Toolkit (JATE)2, where we originally implemented five state-of-the-art algorithms under a uniform framework (Zhang et al., 2008). This work makes one step further 1Original link: http://lcl2.di.uniroma1.it/ 2https://code.google.com/p/jatetoolkit. Google Code has …

Content truncated to 500 words