Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts Elizabeth Clark1∗ Asli Celikyilmaz2 Noah A. Smith1,3 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Microsoft Research 3Allen Institute for Artificial Intelligence {eaclark7,nasmith}@cs.washington.edu aslicel@microsoft.com Abstract For evaluating machine-generated texts, au- tomatic methods hold the promise of avoid- ing collection of human judgments, which can be expensive and time-consuming. The most common automatic metrics, like BLEU and ROUGE, depend on exact word match- ing, an inflexible approach for measuring se- mantic similarity. We introduce methods based on sentence mover’s similarity; our au- tomatic metrics evaluate text in a continuous space using word and sentence embeddings. We find that sentence-based metrics corre- late with human judgments significantly bet- ter than ROUGE, both on machine-generated summaries (average length of 3.4 sentences) and human-authored essays (average length of 7.5). We also show that sentence mover’s sim- ilarity can be used as a reward when learning a generation model via reinforcement learning; we present both automatic and human evalua- tions of summaries learned in this way, finding that our approach outperforms ROUGE. 1 Introduction Automatic text evaluation reduces the need for human evaluations, which can be expensive and time-consuming to collect, particularly when eval- uating long, multi-sentence texts. Automatic met- rics allow faster measures of progress when train- ing and testing models and easier development of text generation systems. However, existing automatic metrics for evalu- ating text are problematic. Due to their computa- tional efficiency, metrics based on word-matching are common, such as ROUGE (Lin, 2004) for summarization, BLEU (Papineni et al., 2002) for machine translation, and METEOR (Banerjee and Lavie, 2005) or CIDER (Vedantam et al., 2015) for image captioning. Nevertheless, these metrics of- ∗ Work done while author was at Microsoft Research. The children eat lunch and play in the park. The family is on a picnic. They have fun. A: B: 3.7 6.3 5.1 6.2 7.6 5.5 6.1 5.1 S+WMS: 5.13 Figure 1: An illustration of S+WMS (a sentence mover similarity metric that uses both word and sentence em- beddings) between two documents. This metric finds the minimal cost of “moving” both the word embed- dings (orange) and the sentence embeddings (blue) in Document A to those in Document B. An arrow’s width is the proportion of the embedding’s weight be- ing moved, and its label is the Euclidean distance. Here we show only the highest weighted connections. ten fail to capture information that has been re- worded or reordered from the reference text, as shown in Kilickaya et al. (2017) and Table 1.1 They have also been found to correlate weakly with human judgments (Liu et al., 2016; Novikova et al., 2017). To avoid these shortcomings, word mover’s dis- tance (WMD; Kusner et al., 2015) can be used to evaluate text in a continuous space using pre- trained word embeddings instead of relying on exact word matching. WMD has been used suc- cessfully for tasks including image caption eval- uation (Kilickaya et al., 2017), automatic essay evaluation (Tashu and …

Content truncated to 500 words