A confidence-based framework for disambiguating geographic terms

A confidence-based framework for disambiguating geographic terms Erik Rauch, Michael Bukatin, and Kenneth Baker MetaCarta, Inc. 875 Massachusetts Avenue, 6th Floor Cambridge, MA 02139 {rauch,mishka,bakerkj}@metacarta.com Abstract We describe a purely confidence-based geo- graphic term disambiguation system that cru- cially relies on the notion of “positive” and “negative” context and methods for combining confidence-based disambiguation with mea- sures of relevance to a user’s query. 1 Introduction Many questions about geographic term disambiguation are standardly handled in a statistical framework: for ex- ample, we can ask that, in the absence of contextual in- formation, with what probability does the word Madison refer to a person (e.g. James Madison), an organization (e.g. Madison Guaranty Savings and Loan), or a place (e.g. Madison, Wisconsin), and if no other disambigua- tion alternative exists, we can expect these three numbers to sum to 1 (i.e. behave like probabilities). However, there are many other questions where a strictly probability-based framework is less appropriate. In particular, much of the information that could be used to disambiguate spatial references in natural language text is strongly non-local in character, and as we increase the amount of this background information, eventually we reach the point when the amount of training data per parameter is so low that there is no repeatable experiment to base probabilities on. In such cases, “probabilities” are effectively used as a stand-in for what is really our confidence in one judgment or another. In this paper we describe some of the meth- ods used in a purely confidence-based geographic term disambiguation system that crucially relies on the notion of “positive” and “negative” context. Far more information is contained in unstructured text (such as the Web and message traffic) than in structured databases, so automatically processing ambiguous geo- graphic references unlocks a large amount of informa- Figure 1: MetaCarta Geographic Text Search interface, showing query results ranked and plotted on a map. tion. Adding spatial dimensions to the document search systems requires new algorithms for determining the rele- vance of documents. We describe methods for combining confidence-based disambiguation with measures of rele- vance to a user’s query. It has become clear after several decades of artificial intelligence research that automated general natural lan- guage understanding is not feasible yet. However, we have been able to make progress by restricting our ef- fort to the well-defined domain of geographic concepts, using statistical methods on extremely large corpora. To cope with billions of documents, we have built fast al- gorithms for extracting and disambiguating geographic Figure 2: The distribution of occurrences of a term can identify geographic areas that it is relevant to. This ex- ample shows the distribution of the word wine in Europe. information and fast database algorithms specifically for information which has a spatial component. One form of information retrieval made possible by extracting geographic meaning in large corpora is geo- graphic text search. Users are presented with an interface containing a traditional text search form combined with a map. They can zoom in on areas of …

Content truncated to 500 words