Discriminative Learning over Constrained Latent Representations

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 429–437, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Discriminative Learning over Constrained Latent Representations Ming-Wei Chang and Dan Goldwasser and Dan Roth and Vivek Srikumar University of Illinois at Urbana Champaign Urbana, IL 61801 {mchang,goldwas1,danr,vsrikum2}@uiuc.edu Abstract This paper proposes a general learning frame- work for a class of problems that require learn- ing over latent intermediate representations. Many natural language processing (NLP) de- cision problems are defined over an expressive intermediate representation that is not explicit in the input, leaving the algorithm with both the task of recovering a good intermediate rep- resentation and learning to classify correctly. Most current systems separate the learning problem into two stages by solving the first step of recovering the intermediate representa- tion heuristically and using it to learn the final classifier. This paper develops a novel joint learning algorithm for both tasks, that uses the final prediction to guide the selection of the best intermediate representation. We evalu- ate our algorithm on three different NLP tasks – transliteration, paraphrase identification and textual entailment – and show that our joint method significantly improves performance. 1 Introduction Many NLP tasks can be phrased as decision prob- lems over complex linguistic structures. Successful learning depends on correctly encoding these (of- ten latent) structures as features for the learning sys- tem. Tasks such as transliteration discovery (Kle- mentiev and Roth, 2008), recognizing textual en- tailment (RTE) (Dagan et al., 2006) and paraphrase identification (Dolan et al., 2004) are a few proto- typical examples. However, the input to such prob- lems does not specify the latent structures and the problem is defined in terms of surface forms only. Most current solutions transform the raw input into a meaningful intermediate representation1, and then encode its structural properties as features for the learning algorithm. Consider the RTE task of identifying whether the meaning of a short text snippet (called the hypoth- esis) can be inferred from that of another snippet (called the text). A common solution (MacCartney et al., 2008; Roth et al., 2009) is to begin by defining an alignment over the corresponding entities, pred- icates and their arguments as an intermediate rep- resentation. A classifier is then trained using fea- tures extracted from the intermediate representation. The idea of using a intermediate representation also occurs frequently in other NLP tasks (Bergsma and Kondrak, 2007; Qiu et al., 2006). While the importance of finding a good inter- mediate representation is clear, emphasis is typi- cally placed on the later stage of extracting features over this intermediate representation, thus separat- ing learning into two stages – specifying the la- tent representation, and then extracting features for learning. The latent representation is obtained by an inference process using predefined models or well- designed heuristics. While these approaches often perform well, they ignore a useful resource when generating the latent structure – the labeled data for the final learning task. As we will …

Content truncated to 500 words