Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues Using BERT

Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT Ye Liu1, Wolfgang Maier2, Wolfgang Minker3 and Stefan Ultes4 1,2,4Mercedes-Benz AG, Sindelfingen, Germany 1,3Ulm University, Ulm, Germany 1ye.y.liu@daimler.com 2,4{wolfgang.mw.maier,stefan.ultes}@daimler.com 3wolfgang.minker@uni-ulm.de Abstract This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expen- sive and time-consuming human labor, we present this novel task of automatic natural- ness evaluation of generated language. By fine-tuning the BERT model, our proposed nat- uralness evaluation method shows robust re- sults and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge. 1 Introduction With the increasing popularity of virtual assistants such as Alexa or Siri, users have a higher demand for conducting natural conversations. They would like to chat with these assistants more naturally— maybe even like talking to a real human being. One of the key questions arising from this, though, is how to measure the naturalness of the generated language. In the past, native speakers judged the quality of the generated language by answering questions like “is this utterance natural?” or “could it have been produced by a native speaker?” to rate the naturalness (Novikova et al., 2016). However, this approach heavily depends on manual effort and is impractical for broader use. On the other hand, the widely used automatic metrics for evaluating language generation, like BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005), rely on word overlap mechanism and compare the generated sentence to one or more human-created reference sentences (Stent et al., 2005), but cannot directly reflect naturalness information. Likert scale ratings are generally used in human evaluation. The Figure 1 shows an example of Sentence: X is a moderately priced restaurant in X. very unnatural ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ very natural Figure 1: Example of 6-point Likert scale for evaluat- ing the naturalness of a sentence. 6-point Likert scale rating on naturalness of a sen- tence. In general, the human evaluation items (nat- uralness, coherence, etc.) are hard explainable ob- jectively and native speakers purely rely on their underlying criteria to judge the performance of gen- erated language without any specified rules. To alle- viate the manual effort previously needed, the goal of this paper is to render the task of assessing the naturalness of generated language as an automatic reference-based classification problem having each Likert scale rating representing one class. To our knowledge, we are the first to propose this task and present an approach evaluating naturalness of gen- eration through fine-tuning a pre-trained Language Model (LM). The contribution of this paper is two-fold: a pre- trained BERT model is fine-tuned to estimate the naturalness of a generated sentence based on a refer- ence sentence. Three baselines are proposed for ro- bust performance comparison: a support vector ma- chine (SVM) baseline using bag-of-words (BoW) vectors for input …

Content truncated to 500 words