There’s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2109–2115, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics There’s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction Courtney Napoles,1 Keisuke Sakaguchi,1 and Joel Tetreault2 1Center for Language and Speech Processing, Johns Hopkins University 2Grammarly {napoles,keisuke}@cs.jhu.edu, joel.tetreault@grammarly.com Abstract Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing gram- matical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metrics. By interpolating both methods, we achieve state-of-the-art correlation with hu- man judgments. Finally, we show that GEC metrics are much more reliable when they are calculated at the sentence level instead of the corpus level. We have set up a CodaLab site for benchmarking GEC output using a com- mon dataset and different evaluation metrics. 1 Introduction Grammatical error correction (GEC) has been evalu- ated by comparing the changes made by a system to the corrections made in gold-standard annotations. Following the recent shared tasks in this field (e.g., Ng et al. (2014)), several papers have critiqued GEC metrics and proposed new methods. Existing met- rics depend on gold-standard corrections and there- fore have a notable weakness: systems are penalized for making corrections that do not appear in the ref- erences.1 For example, the following output has low metric scores even though three appropriate correc- tions were made to the input: 1We refer to the gold-standard corrections as references be- cause gold standard suggests just one accurate correction. However , people now can contact communicate with anyone people all over the world who can use computers at any time , and there is no time delay of messages . These changes (in red) were not seen in the refer- ences and therefore the metrics GLEU and M2 (de- scribed in §2) score this output worse than 75% of 15,000 other generated sentences. While grammaticality-based, reference-less met- rics have been effective in estimating the quality of machine translation (MT) output, the utility of such metrics has not been investigated previously for GEC. We hypothesize that such methods can over- come this weakness in reference-based GEC met- rics. This paper has four contributions: 1) We de- velop three grammaticality metrics that are com- petitive with current reference-based measures and correlate very strongly with human judgments. 2) We achieve state-of-the-art performance when in- terpolating a leading reference-based metric with a grammaticality metric. 3) We identify an interest- ing result that the mean of sentence-level scores is substantially better for evaluating systems than the system-level score. 4) We release code for two grammaticality metrics and establish an online plat- form for evaluating GEC output. 2 Prior work To our knowledge, this is the first work to evalu- ate GEC without references. Within MT, this task is called quality estimation. MT output is evalu- ated by its fluency, …

Content truncated to 500 words