Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 423–431, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Who, What, When, Where, Why? Comparing Multiple Approaches to the Cross-Lingual 5W Task Kristen Parton*, Kathleen R. McKeown*, Bob Coyne*, Mona T. Diab*, Ralph Grishman†, Dilek Hakkani-Tür‡, Mary Harper§, Heng Ji•, Wei Yun Ma*, Adam Meyers†, Sara Stolbach*, Ang Sun†, Gokhan Tur˚, Wei Xu† and Sibel Yaman‡ *Columbia University New York, NY, USA {kristen, kathy, coyne, mdiab, ma, sara}@cs.columbia.edu †New York University New York, NY, USA {grishman, meyers, asun, xuwei} @cs.nyu.edu ‡International Computer Science Institute Berkeley, CA, USA {dilek, sibel} @icsi.berkeley.edu §Human Lang. Tech. Ctr. of Excellence, Johns Hopkins and U. of Maryland, College Park mharper@umd.edu •City University of New York New York, NY, USA hengji@cs.qc.cuny.edu ˚SRI International Palo Alto, CA, USA gokhan@speech.sri.com Abstract Cross-lingual tasks are especially difficult due to the compounding effect of errors in language processing and errors in machine translation (MT). In this paper, we present an error analysis of a new cross-lingual task: the 5W task, a sentence-level understanding task which seeks to return the English 5W's (Who, What, When, Where and Why) corresponding to a Chinese sentence. We analyze systems that we developed, identifying specific prob- lems in language processing and MT that cause errors. The best cross-lingual 5W sys- tem was still 19% worse than the best mono- lingual 5W system, which shows that MT significantly degrades sentence-level under- standing. Neither source-language nor target- language analysis was able to circumvent problems in MT, although each approach had advantages relative to the other. A detailed error analysis across multiple systems sug- gests directions for future research on the problem. 1 Introduction In our increasingly global world, it is ever more likely for a mono-lingual speaker to require in- formation that is only available in a foreign lan- guage document. Cross-lingual applications ad- dress this need by presenting information in the speaker’s language even when it originally ap- peared in some other language, using machine translation (MT) in the process. In this paper, we present an evaluation and error analysis of a cross-lingual application that we developed for a government-sponsored evaluation, the 5W task. The 5W task seeks to summarize the informa- tion in a natural language sentence by distilling it into the answers to the 5W questions: Who, What, When, Where and Why. To solve this problem, a number of different problems in NLP must be addressed: predicate identification, ar- gument extraction, attachment disambiguation, location and time expression recognition, and (partial) semantic role labeling. In this paper, we address the cross-lingual 5W task: given a source-language sentence, return the 5W’s trans- lated (comprehensibly) into the target language. Success in this task requires a synergy of suc- cessful MT and answer selection. The questions we address in this paper are: • How much does machine translation (MT) degrade the performance of cross-lingual 5W systems, as compared to monolingual performance? • Is it better to do source-language analysis and …

Content truncated to 500 words