Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

Findings of the Association for Computational Linguistics: ACL 2022, pages 150 - 157 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics Jiannan Xiang♥∗, Huayang Li♠, Yahui Liu⋆, Lemao Liu♣, Guoping Huang♣, Defu Lian♦, Shuming Shi♣ ♥Carnegie Mellon University, ♠Nara Institute of Science and Technology ♦University of Science and Techology of China ⋆University of Trento, Italy, ♣Tencent AI Lab jiannanx@cs.cmu.edu, li.huayang.lh6@is.naist.jp yahui.liu@unitn.it, lemaoliu@gmail.com, {donkeyhuang, shumingshi}@tencent.com Abstract Current practices in metric evaluation focus on one single dataset, e.g., Newstest dataset in each year’s WMT Metrics Shared Task. How- ever, in this paper, we qualitatively and quan- titatively show that the performances of met- rics are sensitive to data. The ranking of met- rics varies when the evaluation is conducted on different datasets. Then this paper further investigates two potential hypotheses, i.e., in- significant data points and the deviation of In- dependent and Identically Distributed (i.i.d) as- sumption, which may take responsibility for the issue of data variance. In conclusion, our findings suggest that when evaluating auto- matic translation metrics, researchers should take data variance into account and be cautious to claim the result on a single dataset, because it may leads to inconsistent results with most of other datasets. 1 Introduction Assessing the quality of machine translation (MT) systems is always crucial to promote MT re- search (Marie et al., 2021). Since it is costly and time-consuming for human graders to evaluate ma- chine translation (MT) systems, designing auto- matic metrics for MT has drawn booming attention during the past decades, and many metrics such as BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) have been proposed consequently. Generally, it is non-trivial to measure auto- matic metrics. Conference Machine Translation (WMT) (Ma et al., 2019, 2018; Mach´aˇcek and Bo- jar, 2013a,b; Bojar et al., 2016) thereby holds the Metric Shared Task to evaluate the performance of automatic metrics. In each year, WMT organizers collect a dataset consisting of many MT outputs annotated with human judgments, and automatic metrics are evaluated on the dataset in terms of ∗Work done while J. Xiang was an intern at Tencent AI Lab. their correlations to human judgments. Over the past ten years, the official evaluation reports only independently analyzed the results of that year. To the best of our knowledge, there are no studies to put the evaluation results of ten years together and make a more systematic analysis. Therefore, some key questions remain unknown: are the evaluation results consistent across different years? Are the results on each dataset reliable? One may simply summarize the existing results from the official evaluation reports of the past years and answer the above questions accordingly. How- ever, the existing results use Pearson’s correlation for evaluation which suffers from sensitivity to out- lier data points as argued by Mathur et al. (2020). Besides, involved metrics in the evaluation are dif- ferent year by year, thus it is difficult to directly compare the results among different years. To …

Content truncated to 500 words