Macro-Average: Rare Types Are Important Too

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1138–1157 June 6–11, 2021. ©2021 Association for Computational Linguistics 1138 Macro-Average: Rare Types Are Important Too Thamme Gowda Information Sciences Institute University of Southern California tg@isi.edu Weiqiu You Dept of Computer and Information Science University of Pennsylvania weiqiuy@seas.upenn.edu Constantine Lignos Michtom School of Computer Science Brandeis University lignos@brandeis.edu Jonathan May Information Sciences Institute University of Southern California jonmay@isi.edu Abstract While traditional corpus-level evaluation met- rics for machine translation (MT) correlate well with fluency, they struggle to reflect ad- equacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong cor- relation results. These models, however, re- quire potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We ex- plore the simple type-based classifier metric, MACROF1, and study its applicability to MT evaluation. We find that MACROF1 is com- petitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Fur- ther, we show that MACROF1 can be used to effectively compare supervised and unsuper- vised neural machine translation, and reveal significant qualitative differences in the meth- ods’ outputs.1 1 Introduction Model-based metrics for evaluating machine trans- lation such as BLEURT (Sellam et al., 2020), ESIM (Mathur et al., 2019), and YiSi (Lo, 2019) have re- cently attracted attention due to their superior cor- relation with human judgments (Ma et al., 2019). However, BLEU (Papineni et al., 2002) remains the most widely used corpus-level MT metric. It corre- lates reasonably well with human judgments, and moreover is easy to understand and cheap to cal- culate, requiring only reference translations in the target language. By contrast, model-based metrics require tuning on thousands of examples of human evaluation for every new target language or domain 1Tools and analysis are available at https://github. com/thammegowda/007-mt-eval-macro. MT evaluation met- rics are at https://github.com/isi-nlp/sacrebleu/tree/ macroavg-naacl21. (Sellam et al., 2020). Model-based metric scores are also opaque and can hide undesirable biases, as can be seen in Table 1. Reference: You must be a doctor. Hypothesis: must be a doctor. He -0.735 Joe -0.975 Sue -1.043 She -1.100 Reference: It is the greatest country in the world. Hypothesis: Franceis the greatest country in the world. -0.022 America -0.060 Russia -0.161 Canada -0.309 Table 1: A demonstration of BLEURT’s internal bi- ases; model-free metrics like BLEU would consider each of the errors above to be equally wrong. The source of model-based metrics’ (e.g. BLEURT) correlative superiority over model-free metrics (e.g. BLEU) appears to be the former’s ability to focus evaluation on adequacy, while the latter are overly focused on fluency. BLEU and most other generation metrics consider each output token equally. Since natural language is dominated by a few high-count types, an MT model that con- centrates on getting its ifs, ands and buts right will benefit from BLEU in the long run more than one that gets …

Content truncated to 500 words