Adaptation of Maximum Entropy Capitalizer: Little Data Can Help a Lo

Adaptation of Maximum Entropy Capitalizer: Little Data Can Help a Lot Ciprian Chelba and Alex Acero Microsoft Research One Microsoft Way Redmond, WA 98052 {chelba,alexac}@microsoft.com Abstract A novel technique for maximum “a posteriori” (MAP) adaptation of maximum entropy (MaxEnt) and maximum entropy Markov models (MEMM) is presented. The technique is applied to the problem of recov- ering the correct capitalization of uniformly cased text: a “background” capitalizer trained on 20Mwds of Wall Street Journal (WSJ) text from 1987 is adapted to two Broadcast News (BN) test sets — one containing ABC Primetime Live text and the other NPR Morning News/CNN Morning Edition text — from 1996. The “in-domain” performance of the WSJ capi- talizer is 45% better than that of the 1-gram base- line, when evaluated on a test set drawn from WSJ 1994. When evaluating on the mismatched “out-of- domain” test data, the 1-gram baseline is outper- formed by 60%; the improvement brought by the adaptation technique using a very small amount of matched BN data — 25-70kwds — is about 20-25% relative. Overall, automatic capitalization error rate of 1.4% is achieved on BN data. 1 Introduction Automatic capitalization is a practically relevant problem: speech recognition output needs to be capitalized; also, modern word processors perform capitalization among other text proofing algorithms such as spelling correction and grammar checking. Capitalization can be also used as a preprocessing step in named entity extraction or machine trans- lation. We study the impact of using increasing amounts of training data as well as using a small amount of adaptation data on this simple problem that is well suited to data-driven approaches since vast amounts of “training” data are easily obtainable by simply wiping the case information in text. As in previous approaches, the problem is framed as an instance of the class of sequence labeling problems. A case frequently encountered in prac- tice is that of using mismatched — out-of-domain, in this particular case we used Broadcast News — test data. For example, one may wish to use a capi- talization engine developed on newswire text for e- mail or office documents. This typically affects neg- atively the performance of a given model, and more sophisticated models tend to be more brittle. In the capitalization case we have studied, the relative per- formance improvement of the MEMM capitalizer over the 1-gram baseline drops from in-domain — WSJ — performance of 45% to 35-40% when used on the slightly mismatched BN data. In order to take advantage of the adaptation data in our scenario, a maximum a-posteriori (MAP) adaptation technique for maximum entropy (Max- Ent) models is developed. The adaptation procedure proves to be quite effective in further reducing the capitalization error of the WSJ MEMM capitalizer on BN test data. It is also quite general and could improve performance of MaxEnt models in any sce- nario where model adaptation is desirable. A further relative improvement of about 20% is obtained by adapting the WSJ model to Broadcast News (BN) text. Overall, the …

Content truncated to 500 words