On building a high performance gazetteer database

On building a high performance gazetteer database Amittai E. Axelrod MetaCarta, Inc. 875 Massachusetts Ave., 6th Flr. Cambridge, MA, 02139 amittai@metacarta.com Abstract We define a data model for storing geographic information from multiple sources that en- ables the efficient production of customizable gazetteers. The GazDB separates names from features while storing the relationships be- tween them. Geographic names are stored in a variety of resolutions to allow for i18n and for multiplicity of naming. Geographic fea- tures are categorized along several axes to fa- cilitate selection and filtering. 1 Introduction We are interested in collecting the largest possible set of geographic entities, so as to be able to produce a variety of extremely comprehensive gazetteers. These gazetteers are currently produced to search for both direct and indi- rect geospatial references in text. The production process can be tailored to produce custom gazetteers for other ap- plications, such as historical queries. The purpose of the MetaCarta GazDB is to provide both a place and supporting mechanisms for storing, maintaining, and exporting everything we know about our collection of geographic entities. To produce a gazetteer from various data sources, we make use of a database, the GazDB, as well as two sets of scripts: conversion scripts, to transfer the data from its source format into the GazDB, and export scripts to output data from the GazDB in the form of gazetteers. The interaction between these elements is illustrated in Figure 1. Geographic input data is collected from multiple (not necessarily disjoint) sources, each with their own pecu- liar format. As such, the conversion scripts must perform some amount of normalization and classification of the input data in order to maintain a single unified repository Figure 1: The gazetteer production process of geographic data. However, in order to justify the over- head of consolidating all the data into a single entity, it must be possible to output all of it into multiple gazetteers designed for different goals. It should also be possible to perform filtering oper- ations on the gazetteer entries, such as comparing en- try names against common-language dictionaries. This can be used determine whether occurrences of gazetteer names in documents are geographically relevant (Rauch et al., 2003). This is the task for the export scripts. However, in this paper, we shall focus on the heart of the system, namely the GazDB. Section 2 describes how the GazDB relates geographic names and features. In Section 3 we describe how the GazDB handles ambiguities and inconsistencies in geographic names. Finally, in Section 4 we outline the classification and storage system used for geographic features. 2 Gazetteer entries in the GazDB The most basic form of a gazetteer entry consists of a mapping between a geographic name and a geographic Figure 2: Relating features and names in the GazDB location. The Alexandria Digital Library Project (Hill, 2000), however, defines a gazetteer entry as also requir- ing a type designation to describe the entity referred to by the name and location. Because a geographical type …

Content truncated to 500 words