Natural Language Generation through Character-based RNNs with Finite-state Prior Knowledge

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1083–1092, Osaka, Japan, December 11-17 2016. Natural Language Generation through Character-Based RNNs with Finite-State Prior Knowledge Raghav Goyal∗ Twenty Billion Neurons Berlin, Germany raghav.goyal@twentybn.com Marc Dymetman Xerox Research Centre Europe Grenoble, France marc.dymetman@xerox.com Eric Gaussier LIG, Uni. Grenoble Alpes Grenoble, France eric.gaussier@imag.fr Abstract Recently Wen et al. (2015) have proposed a Recurrent Neural Network (RNN) approach to the generation of utterances from dialog acts, and shown that although their model requires less ef- fort to develop than a rule-based system, it is able to improve certain aspects of the utterances, in particular their naturalness. However their system employs generation at the word-level, which requires one to pre-process the data by substituting named entities with placeholders. This pre- processing prevents the model from handling some contextual effects and from managing multi- ple occurrences of the same attribute. Our approach uses a character-level model, which unlike the word-level model makes it possible to learn to “copy” information from the dialog act to the target without having to pre-process the input. In order to avoid generating non-words and inventing information not present in the input, we propose a method for incorporating prior knowledge into the RNN in the form of a weighted finite-state automaton over character sequences. Automatic and human evaluations show improved performance over baselines on several evaluation criteria. 1 Introduction Rule-based Natural Language Generation systems (Reiter and Dale, 2000) have been quite successful but they suffer from some limitations. They require extensive human effort and tend to produce fixed, repetitive outputs, which do not closely match human-like utterances. For this reason, there has been much interest recently in developing NLG systems which are, at least partially, able to learn from human produced data (Langkilde and Knight, 1998; Belz, 2008). In the last couple of years, Neural Network (NN) based approaches have gained enormous popularity within statistical NLP generally, with applications to Machine Translation (Sutskever et al., 2014), Con- versation Modelling (Vinyals and Le, 2015) and Parsing (Tai et al., 2015), to cite only a few. In particular, architectures based on Recurrent Neural Network (RNN) such as LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Cho et al., 2014) have been succesfully used in Language Modelling tasks due to their ability to model sequential information with long-range dependencies. An RNN-based approach to NLG has been recently proposed by Wen et al. (2015) in the context of a dialog system, where the input semantic representation is a Dialog Act (DA). The decoder uses words as the units of generation. This word-based model requires pre-processing the original data by substituting named entities with placeholders, a process called de-lexicalisation. This is necessary because a standard word-level RNN is not able to “copy” input entities into the target, but has to learn each correspondence individually, which it can only do with a lot of data. Such a de-lexicalization approach has the advantage of reducing data sparsity, but it also suffers from various shortcomings: …

Content truncated to 500 words