Model Selection for Type-Supervised Learning with Application to POS Tagging

Proceedings of the 19th Conference on Computational Language Learning, pages 332–337, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Model Selection for Type-Supervised Learning with Application to POS Tagging Kristina Toutanova Microsoft Research Redmond, WA, USA Waleed Ammar∗ School of Computer Science Carnegie Mellon University Pallavi Chourdhury Hoifung Poon Microsoft Research Redmond, WA, USA Abstract Model selection (picking, for example, the feature set and the regularization strength) is crucial for building high-accuracy NLP models. In supervised learning, we can es- timate the accuracy of a model on a subset of the labeled data and choose the model with the highest accuracy. In contrast, here we focus on type-supervised learn- ing, which uses constraints over the pos- sible labels for word types for supervi- sion, and labeled data is either not avail- able or very small. For the setting where no labeled data is available, we perform a comparative study of previously proposed and one novel model selection criterion on type-supervised POS-tagging in nine lan- guages. For the setting where a small la- beled set is available, we show that the set should be used for semi-supervised learn- ing rather than for model selection only – using it for model selection reduces the er- ror by less than 5%, whereas using it for semi-supervised learning reduces the error by 44%. 1 Introduction Fully supervised training of NLP models (e.g., part-of-speech taggers, named entity recognizers, relation extractors) works well when plenty of la- beled examples are available. However, manu- ally labeled corpora are expensive to construct in many languages and domains, whereas an al- ternative, if weaker, supervision is often read- ily available. For example, corpora labeled with POS tags at the token level are only available for around 35 languages, while tag dictionaries of the ∗This research was conducted during the author’s intern- ship at Microsoft Research. form displayed in Fig. 1 are available for many more languages, either in commercial dictionar- ies or community created resources such as Wik- tionary. Tag dictionaries provide type-level super- vision for word types in the lexicon. Similarly, while sentences labeled with named entities are scarce, gazetteers and databases are more readily available (Bollacker et al., 2008). There has been substantial research on how best to build models using such type-level supervision, for POS tagging, super sense tagging, NER, and relation extraction (Craven et al., 1999; Smith and Eisner, 2005; Carlson et al., 2009; Mintz et al., 2009; Johannsen et al., 2014), inter alia, focussing on parametric forms and loss functions for model training. However, there has been little research on the practically important aspect of model selection for type-supervised learning. While some previ- ous work used criteria based on the type-level su- pervision only (Smith and Eisner, 2005; Goldwa- ter and Griffiths, 2007), much prior work used a la- beled set for model selection (Vaswani et al., 2010; Soderland and Weld, 2014). We are not aware of any prior work aiming to compare or improve ex- isting type-supervised model selection criteria. For …

Content truncated to 500 words