An Online Algorithm for Learning over Constrained Latent Representations using Multiple Views

International Joint Conference on Natural Language Processing, pages 1072–1076, Nagoya, Japan, 14-18 October 2013. An Online Algorithm for Learning over Constrained Latent Representations using Multiple Views ∗ Ann Clifton and Max Whitney and Anoop Sarkar School of Computing Science Simon Fraser University 8888 University Drive Burnaby BC. V5A 1S6. Canada {ann clifton, mwhitney, anoop}@sfu.ca Abstract We introduce an online framework for dis- criminative learning problems over hidden structures, where we learn both the latent structure and the classifier for a supervised learning task. Previous work on lever- aging latent representations for discrimi- native learners has used batch algorithms that require multiple passes though the en- tire training data. Instead, we propose an online algorithm that efficiently jointly learns the latent structures and the classi- fier. We further extend this to include mul- tiple views on the latent structures with different representations. Our proposed online algorithm with multiple views sig- nificantly outperforms batch learning for latent representations with a single view on a grammaticality prediction task. 1 Introduction Natural language data is implicitly richly struc- tured, and making use of that structure can be valuable in a wide variety of NLP tasks. How- ever, finding these latent structures is a complex task of its own right. Early work used a two- phase pipeline process, in which the output of a structure prediction algorithm (e.g. a noun phrase finder) acts as fixed input features to train a classi- fier for a different task (e.g. grammaticality predic- tion). Chang et al. (2009), Das and Smith (2009), Goldwasser and Roth (2008), and Mccallum and Bellare (2005) have shown that this approach can propagate error from the structured prediction to the task-specific classifier. Recent work has com- bined unsupervised learning of (latent) structure prediction with a supervised learning approach for the task. Work in this vein has focused on jointly ∗This research was partially supported by an NSERC, Canada (RGPIN: 264905) grant and a Google Faculty Award to the third author. learning the latent structures together with the task-specific classifier (Cherry and Quirk, 2008; Chang et al., 2010). Chang et al. (2010) in partic- ular introduce a framework for solving classifica- tion problems using constraints over latent struc- tures, referred to as Learning over Constrained Latent Representations (LCLR). We extend this framework for discriminative joint learning over latent structures to a novel online algorithm. Our algorithm learns the latent structures in an unsu- pervised manner, but it can be initialized with the model weights from a supervised learner for the latent task trained on some (other) annotated data. This can be seen as a form of domain adaptation from the supervised latent structure training data to the different classification task. We evaluate our algorithm in comparison to the LCLR batch method on a grammaticality test using a discriminative model that learns shal- low parse (chunk) structures. Our online method has standard convergence guarantees for a max- margin learner, but attains higher accuracy. Fur- thermore, in practice we find that it requires fewer passes over the …

Content truncated to 500 words