Generation that Exploits Corpus-Based Statistical Knowledge

Generation that Exploits Corpus-Based Statistical Knowledge Irene Langkilde and Kevin Knight Information Sciences Institute University of Southern California Marina del Rey, CA 90292 ilangkil@isi, edu and knight@isi, edu Abstract We describe novel aspects of a new natural lan- guage generator called Nitrogen. This generator has a highly flexible input representation that allows a spectrum of input from syntactic to semantic depth, and shifts' the burden of many linguistic decisions to the statistical post-processor. The generation al- gorithm is compositional, making it efficient, yet it also handles non-compositional aspects of language. Nitrogen's design makes it robust and scalable, op- erating with lexicons and knowledge bases of one hundred thousand entities. 1 Introduction Language generation is an important subtask of applications like machine translation, human- computer dialogue, explanation, and summariza- tion. The recurring need for generation suggests the usefulness of a general-purpose, domain-independent natural language generator (NLG). However, "plug- in" generators available today, such as FUF/SURGE (Elhadad and Robin, 1998), MUMBLE (Meteer et al., 1987), KPML (Bateman, 1996), and CoGen- Tex's RealPro (Lavoie and Rambow, 1997), require inputs with a daunting amount of linguistic detail. As a result, many client applications resort instead to simpler template-based methods. An important advantage of templates is that they sidestep linguistic decision-making, and avoid the need for large complex knowledge resources and pro- cessing. For example, the following structure could be a typical result from a database query on the type of food a venue serves: ((:obj-type venue)(:obj-name Top_of_the_Mark) (:attribute food-type)(:attrib-value American)) By using a template like <obj-name> 's <attribute> is <attrib-value>. the structure could produce the sentence, "Top of the Mark's food type is American." .Templates avoid the need for detailed linguistic information about lexical items, part-of-speech tags, number, gender, definiteness, tense, sentence organi- zation, sub-categorization structure, semantic rela- • tions, etc., that more general NLG methods need to have specified in the input (or supply defaults for). Such information is usually not readily inferrable from an application's database, nor is it always read- ily available from other sources, with the breadth of coverage or level of detail that is needed. Thus, using a general-purpose generator can be formidable (Re- iter, 1995). However, templates only work in very controlled or limited situations. They cannot pro- vide the expressiveness, flexibility or scalability that many real domains need. A desirable solution is a generator that abstracts away from templates enough to provide the needed flexibility and scalability, and yet still requires only minimal semantic input (and maintains reasonable efficiency). This generator would take on the re- sponsibility of finding an appropriate linguistic re- alization for an underspecified semantic input. This solution is especially important in the context of ma- chine translation, where the surface syntactic orga- nization of the source text is usually different from that of the target language, and the deep semantics are often difficult to obtain or represent completely as well. In Japanese to English translation, for ex- ample, it is often hard to determine from a Japanese text the number or gender of …

Content truncated to 500 words