LaTaT: Language and Text Analysis Tools

LaTaT: Language and Text Analysis Tools Dekang Lin University of Alberta Department of Computing Science Edmonton, Alberta T6H 2E1 Canada lindek@cs.ualberta.ca ABSTRACT LaTaT is a Language and Text Analysis Toolset. This paper gives a brief description of the components comprising LaTaT, including a Minimalist parser and language and concept learning programs. 1. INTRODUCTION In natural language processing, syntactic and semantic knowledge are deeply intertwined with each other, both in their acquisition and usage. The goal of our research is to build a syntactic and semantic knowledge base through an iterative process that involves both language processing and language acquisition. We start the process by parsing a large corpus with a manually constructed parser that has only syntactic knowledge. We then extract lexical semantic and statistical knowledge from the parsed corpus, such as similar words and phrases, collocations and idiomatic expressions, and selectional preferences. In the second cycle, the text corpus is parsed again with the assistance of the newly acquired semantic and statistical knowledge, which allows the parser to better resolve systematic syntactic ambiguities, removing unlikely parts of speech. Our hypothesis is that this will result in higher quality parse trees, which in turn allows extraction of higher quality semantic and statistical knowledge in the second and later cycles. LaTaT is a Language and Text Analysis Toolset that demonstrates this iterative learning process. The main components in the toolset consist of the following: • A broad coverage English parser, called Minipar. The grammar is constructed manually, based on the Minimalist Program (Chomsky 1995). Instead of using a large number of CFG rules, Minipar achieves its broad coverage by using a small set of principles to constrain the overgerating X-bar schema; • A collocation extractor that extracts frequency counts of grammatical dependency relationships from a corpus parsed with Minipar. The frequency counts are then injected into Minipar to help it rank candidate parse trees; • A thesaurus constructor (Lin, 1998) that automatically computes the word similarities based on the distributional characteristics of words in the parsed corpus. The resulting word similarity database can then be used to smooth the probability distribution in statistical language models (Dagan et al, 1997); • A clustering algorithm that constructs Roget-like semantic categories in an unsupervised fashion (Lin and Pantel, 2001a); and • An unsupervised learner to identify similar expressions from a parsed corpus (Lin and Pantel, 2001b). 2. Minipar Minipar is a principle-based English parser (Berwick et al, 1991). Like Principar (Lin, 1993), Minipar represents its grammar as a network where nodes represent grammatical categories and links represent types of syntactic (dependency) relationships. The grammar network consists of 35 nodes and 59 links. Additional nodes and links are created dynamically to represent subcategories of verbs. Minipar employs a message passing algorithm that essentially implements distributed chart parsing. Instead of maintaining a single chart, each node in the grammar network maintains a chart containing partially built structures belonging to the grammatical category represented by the node. The grammatical principles are implemented as constraints associated with the …

Content truncated to 500 words