Weakly-Supervised Bayesian Learning of a CCG Supertagger

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 141–150, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Weakly-Supervised Bayesian Learning of a CCG Supertagger Dan Garrette∗ Chris Dyer† Jason Baldridge‡ Noah A. Smith† ∗Department of Computer Science, The University of Texas at Austin †School of Computer Science, Carnegie Mellon University ‡Department of Linguistics, The University of Texas at Austin ∗Corresponding author: dhg@cs.utexas.edu Abstract We present a Bayesian formulation for weakly-supervised learning of a Combina- tory Categorial Grammar (CCG) supertag- ger with an HMM. We assume supervi- sion in the form of a tag dictionary, and our prior encourages the use of cross- linguistically common category structures as well as transitions between tags that can combine locally according to CCG’s combinators. Our prior is theoretically ap- pealing since it is motivated by language- independent, universal properties of the CCG formalism. Empirically, we show that it yields substantial improvements over previous work that used similar bi- ases to initialize an EM-based learner. Ad- ditional gains are obtained by further shap- ing the prior with corpus-specific informa- tion that is extracted automatically from raw text and a tag dictionary. 1 Introduction Unsupervised part-of-speech (POS) induction is a classic problem in NLP. Many proposed solutions are based on Hidden Markov models (HMMs), with various improvements obtainable through: induc- tive bias in the form of tag dictionaries (Kupiec, 1992; Merialdo, 1994), sparsity constraints (Lee et al., 2010), careful initialization of parameters (Goldberg et al., 2008), feature based represen- tations (Berg-Kirkpatrick et al., 2010; Smith and Eisner, 2005), and priors on model parameters (Johnson, 2007; Goldwater and Griffiths, 2007; Blunsom and Cohn, 2011, inter alia). When tag dictionaries are available, a situa- tion we will call type-supervision, POS induc- tion from unlabeled corpora can be relatively suc- cessful; however, as the number of possible tags increases, performance drops (Ravi and Knight, 2009). In such cases, there are a large number of possible labels for each token, so picking the right one simply by chance is unlikely; the pa- rameter space tends to be large; and devising good initial parameters is difficult. Therefore, it is un- surprising that the unsupervised (or even weakly- supervised) learning of a Combinatory Categorial Grammar (CCG) supertagger, which labels each word with one of a large (possibly unbounded) number of structured categories called supertags, is a considerable challenge. Despite the apparent complexity of the task, su- pertag sequences have regularities due to univer- sal properties of the CCG formalism (§2) that can be used to reduce the complexity of the problem; previous work showed promising results by using these regularities to initialize an HMM that is then refined with EM (Baldridge, 2008). Here, we ex- ploit CCG’s category structure to motivate a novel prior over HMM parameters for use in Bayesian learning (§3). This prior encourages (i) cross- linguistically common tag types, (ii) tag bigrams that can combine using CCG’s combinators, and (iii) sparse transition distributions. We also go be- yond the use of these universals to …

Content truncated to 500 words