Response-based Learning for Machine Translation of Open-domain Database Queries

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1339–1344, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Response-based Learning for Machine Translation of Open-domain Database Queries Carolin Haas Computational Linguistics Heidelberg University 69120 Heidelberg, Germany haas1@cl.uni-heidelberg.de Stefan Riezler Computational Linguistics Heidelberg University 69120 Heidelberg, Germany riezler@cl.uni-heidelberg.de Abstract Response-based learning allows to adapt a sta- tistical machine translation (SMT) system to an extrinsic task by extracting supervision sig- nals from task-specific feedback. In this pa- per, we elicit response signals for SMT adap- tation by executing semantic parses of trans- lated queries against the Freebase database. The challenge of our work lies in scaling se- mantic parsers to the lexical diversity of open- domain databases. We find that parser perfor- mance on incorrect English sentences, which is standardly ignored in parser evaluation, is key in model selection. In our experiments, the biggest improvements in F1-score for re- turning the correct answer from a semantic parse for a translated query are achieved by selecting a parser that is carefully enhanced by paraphrases and synonyms. 1 Introduction In response-based learning for SMT, supervision signals are extracted from an extrinsic response to a machine translation, in contrast to using human- generated reference translations for supervision. We apply this framework to a scenario in which a se- mantic parse of a translated database query is exe- cuted against the Freebase database. We view learn- ing from such task-specific feedback as adaptation of SMT parameters to the task of translating open- domain database queries, thereby grounding SMT in the task of multilingual database access. The success criterion for this task is F1-score in returning the cor- rect answer from a semantic parse of the translated query, rather than BLEU. Since the semantic parser provides feedback to the response-based learner and defines the final evaluation criterion, the challenge of the presented work lies in scaling the seman- tic parser to the lexical diversity of open-domain databases such as Freebase. Riezler et al. (2014) showed how to use response-based learning to adapt an SMT system to a semantic parser for the Geo- query domain. The state-of-the-art in semantic pars- ing on Geoquery achieves a parsing accuracy of over 82% (see Andreas et al. (2013) for an overview), while the state-of-the-art in semantic parsing on the Free917 data (Cai and Yates, 2013) achieves 68.5% accuracy (Berant and Liang, 2014). This is due to the lexical variability of Free917 (2,036 word types) compared to Geoquery (279 word types). In this paper, we compare different ways of scal- ing up state-of-the-art semantic parsers for Freebase by adding synonyms and paraphrases. First, we con- sider Berant and Liang (2014)’s own extension of the semantic parser of Berant et al. (2013) by us- ing paraphrases. Second, we apply WordNet syn- onyms (Miller, 1995) for selected parts of speech to the queries in the Free917 dataset. The new pairs of queries and logical forms are added to the dataset on which …

Content truncated to 500 words