Updating an NLP system to fit new domains: an empirical study on the sentence segmentation problem

Updating an NLP System to Fit New Domains: an empirical study on the sentence segmentation problem Tong Zhang � and Fred Damerau � and David Johnson � IBM T.J. Watson Research Center Yorktown Heights New York, 10598, USA � tzhang@watson.ibm.com � damerau@watson.ibm.com � dejohns@us.ibm.com Abstract Statistical machine learning algorithms have been successfully applied to many natural lan- guage processing (NLP) problems. Compared to manually constructed systems, statistical NLP systems are often easier to develop and maintain since only annotated training text is required. From annotated data, the underlying statistical algorithm can build a model so that annotations for future data can be predicted. However, the performance of a statistical sys- tem can also depend heavily on the character- istics of the training data. If we apply such a system to text with characteristics different from that of the training data, then performance degradation will occur. In this paper, we ex- amine this issue empirically using the sentence boundary detection problem. We propose and compare several methods that can be used to update a statistical NLP system when moving to a different domain. 1 Introduction An important issue for a statistical machine learning based NLP system is that its performance can depend heavily on the characteristics of the training data used to build the system. Consequently if we train a system on some data but apply it to other data with different charac- teristics, then the system’s performance can degrade sig- nificantly. It is therefore natural to investigate the follow- ing related issues: � How to detect the change of underlying data charac- teristics, and to estimate the corresponding system performance degradation. � If performance degradation is detected, how to up- date a statistical system to improve its performance with as little human effort as possible. This paper investigates some methodological and prac- tical aspects of the above issues. Although ideally such a study would include as many different statistical algo- rithms as possible, and as many different linguistic prob- lems as possible (so that a very general conclusion might be drawn), in reality such an undertaking is not only diffi- cult to carry out, but also can hide essential observations and obscure important effects that may depend on many variables. An alternative is to study a relatively simple and well-understood problem to try to gain understand- ing of the fundamental issues. Causal effects and essen- tial observations can be more easily isolated and identi- fied from simple problems since there are fewer variables that can affect the outcome of the experiments. In this paper, we take the second approach and focus on a specific problem using a specific underlying statistical algorithm. However, we try to use only some fundamen- tal properties of the algorithm so that our methods are readily applicable to other systems with similar proper- ties. Specifically, we use the sentence boundary detection problem to perform experiments since not only is it rel- atively simple and well-understood, but it also provides the basis for other more advanced linguistic …

Content truncated to 500 words