Can Semantic Role Labeling Improve SMT?

Proceedings of the 13th Annual Conference of the EAMT, pages 218–225, Barcelona, May 2009 Can Semantic Role Labeling Improve SMT? Dekai WU1 Pascale FUNG2 Human Language Technology Center HKUST 1Department of Computer Science and Engineering 2Department of Electronic and Computer Engineering University of Science and Technology, Clear Water Bay, Hong Kong dekai@cs.ust.hk pascale@ee.ust.hk Abstract We present a series of empirical studies aimed at illuminating more precisely the likely contribution of semantic roles in im- proving statistical machine translation ac- curacy. The experiments reported study several aspects key to success: (1) the fre- quencies of types of SMT errors where semantic parsing and role labeling could help, and (2) if and where semantic roles offer more accurate guidance to SMT than merely syntactic annotation, and (3) the po- tential quantitative impact of realistic se- mantic role guidance to SMT systems, in terms of BLEU and METEOR scores. 1 Introduction In this investigative paper, we present a new set of empirical studies aimed at illuminating more pre- cisely the likely contribution of semantic parsing and role labeling toward improving statistical ma- chine translation accuracy. The most glaring errors made by statistical machine translation systems continue to be those resulting in confusion of semantic roles. These sorts of translation errors often result in serious misunderstandings of the essential meaning of the source utterances — who did what to whom, for whom or what, how, where, when, and why. It has been widely observed that the nega- tive impacts of such errors on the utility of the translation are inadequately reflected by evaluation metrics based on lexical criteria. The accuracy of translation lexical choice has reached increasingly satisfactory levels—at least for largely literal gen- res such as newswire — which helps boost lexi- cally oriented scores such as BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005) c⃝ 2009 European Association for Machine Translation. despite serious role confusion errors in the transla- tions. It has also often been noted that precision- oriented metrics such as BLEU tend to reward flu- ency more than adequacy (in particular, BLEU’s length penalty is only an indirect and weak indica- tor of adequacy). Today’s SMT systems produce translations that often contain significant role con- fusion errors but nevertheless read quite fluently. Thus, while recent years have seen continued improvement in the accuracy of statistical machine translation systems as measured by such lexically based metrics, this underestimates the effect of the persistent errors of role confusion upon the actual translation utility. This situation leads us to consider the poten- tial application of shallow semantic parsing and se- mantic role labeling models to SMT, in ways that might reduce role confusion errors in the transla- tion output. Within the lexical semantics commu- nity, increasingly sophisticated models for shallow semantic parsing are being developed. Such se- mantic parsers, which automatically label the pred- icates and arguments (roles) of the various seman- tic frames in a sentence, could automatically iden- tify inconsistent semantic frame and role mappings between the input …

Content truncated to 500 words