2kenize: Tying Subword Sequences for Chinese Script Conversion

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 681 USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation Shikib Mehri and Maxine Eskenazi Dialog Research Center, Language Technologies Institute Carnegie Mellon University, USA {amehri,max}@cs.cmu.edu Abstract The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language genera- tion metrics have been shown to be ineffec- tive for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains un- supervised models to measure several desir- able qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system- level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally pro- duces interpretable measures for several desir- able properties of dialog. 1 Introduction The lack of meaningful automatic evaluation met- rics is a significant impediment for open-domain dialog generation research. Standard language gen- eration metrics have been shown to be ineffec- tive for dialog evaluation (Deriu et al., 2019; Liu et al., 2016). Without well-accepted, meaningful automatic metrics, open-domain dialog researchers have come to rely on human evaluation. Due to its time- and cost-intensive nature, human eval- uation is typically only used for the final dialog model. As such, during development dialog sys- tems are generally optimized for poorly-correlated automatic metrics (e.g., F-1, BLEU, PPL) which can result in sub-par human evaluation scores (Di- nan et al., 2019). To facilitate development of open- domain dialog models with meaningful automatic metrics, this paper presents the UnSupervised and Reference free (USR) evaluation metric for dialog. Standard automatic metrics for evaluating dialog generation (e.g., BLEU, F-1, METEOR, ROUGE) have several shortcomings that make them unsuit- able for dialog evaluation: (1) The one-to-many nature of dialog (Zhao et al., 2017) makes word- overlap metrics ineffective for scoring valid system output that deviates from the ground-truth response (Liu et al., 2016; Gupta et al., 2019). (2) Human evaluation of dialog typically measures multiple properties (e.g., appropriate, interesting, consis- tent). Automatic metrics on the other hand, con- dense the multi-faceted nature of dialog quality to a single uninterpretable metric. (3) There are many definitions of what a good dialog is and, as such, it is not feasible to construct a “one size fits all” metric. Depending on the task and the data, the desired qualities of a dialog system may differ (Walker et al., 1997; Deriu et al., 2019). USR is a reference-free metric that consists of several interpretable sub-metrics which are com- bined in a configurable manner. Rather than relying on a ground-truth reference response, unsupervised models are trained to measure desired qualities of dialog (e.g., interesting, natural). As such, USR (1) alleviates the one-to-many issue of standard metrics, (2) produces interpretable measures for desirable properties of dialog, and (3) provides a configurable mechanism for combining several sub- metrics into an overall quality …

Content truncated to 500 words