Bayesian Modeling of Lexical Resources for Low-Resource Settings

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1029–1039 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1095 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1029–1039 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1095 Bayesian Modeling of Lexical Resources for Low-Resource Settings Nicholas Andrews and Mark Dredze and Benjamin Van Durme and Jason Eisner Department of Computer Science and Human Language Technology Center of Excellence Johns Hopkins University 3400 N. Charles St., Baltimore, MD 21218 USA {noa,eisner,mdredze,vandurme}@jhu.edu Abstract Lexical resources such as dictionaries and gazetteers are often used as auxiliary data for tasks such as part-of-speech induction and named-entity recognition. However, discriminative training with lexical features requires annotated data to reliably estimate the lexical feature weights and may result in overfitting the lexical features at the ex- pense of features which generalize better. In this paper, we investigate a more robust approach: we stipulate that the lexicon is the result of an assumed generative process. Practically, this means that we may treat the lexical resources as observations under the proposed generative model. The lexi- cal resources provide training data for the generative model without requiring sepa- rate data to estimate lexical feature weights. We evaluate the proposed approach in two settings: part-of-speech induction and low- resource named-entity recognition. 1 Introduction Dictionaries and gazetteers are useful in many natural language processing tasks. These lexical resources may be derived from freely available sources (such as Wikidata and Wiktionary) or con- structed for a particular domain. Lexical resources are typically used to complement existing anno- tations for a given task (Ando and Zhang, 2005; Collobert et al., 2011). In this paper, we focus instead on low-resource settings where task annota- tions are unavailable or scarce. Specifically, we use lexical resources to guide part-of-speech induction (§4) and to bootstrap named-entity recognizers in low-resource languages (§5). Given their success, it is perhaps surprising that incorporating gazetteers or dictionaries into dis- criminative models (e.g. conditional random fields) may sometimes hurt performance. This phenom- ena is called weight under-training, in which lexi- cal features—which detect whether a name is listed in the dictionary or gazetteer—are given exces- sive weight at the expense of other useful features such as spelling features that would generalize to unlisted names (Smith et al., 2005; Sutton et al., 2006; Smith and Osborne, 2006). Furthermore, dis- criminative training with lexical features requires sufficient annotated training data, which poses chal- lenges for the unsupervised and low-resource set- tings we consider here. Our observation is that Bayesian modeling pro- vides a principled solution. The lexicon is itself a dataset that was generated by some process. Prac- tically, this means that lexicon entries (words or phrases) may be treated as additional observations. As a result, these entries provide information about how names are spelled. The presence of the lexi- con therefore now improves training of the spelling features, rather than competing with the spelling …

Content truncated to 500 words