Ensemble of MRR and NDCG models for Visual Dialog

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3272–3363 June 6–11, 2021. ©2021 Association for Computational Linguistics 3272 Ensemble of MRR and NDCG models for Visual Dialog Idan Schwartz Technion NetApp idansc@cs.technion.ac.il Abstract Assessing an AI agent that can converse in hu- man language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over seman- tics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) met- ric evaluates the model performance by tak- ing into account the rank of a single human- derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., ‘yeah’ and ‘yes’). To ad- dress this, the normalized discounted cumula- tive gain (NDCG) metric has been used to cap- ture the relevance of all the correct answers via dense annotations. However, the NDCG met- ric favors the usually applicable uncertain an- swers such as ‘I don’t know.’ Crafting a model that excels on both MRR and NDCG metrics is challenging (Murahari et al., 2020). Ide- ally, an AI agent should answer a human-like reply and validate the correctness of any an- swer. To address this issue, we describe a two- step non-parametric ranking approach that can merge strong MRR and NDCG models. Us- ing our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art perfor- mance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https: //github.com/idansc/mrr-ndcg. 1 Introduction Das et al. (2017) introduced the task of Visual Di- alog, which requires an agent to converse about visual input. Evaluating visually aware conversa- tion should examine both linguistic properties and visual reasoning. Analysis of generative metrics for dialog often shows no correlation with human judgments (Liu et al., 2016). Hence, to evaluate the correctness of the candidate answers, a retrieval approach is preferred. Two metrics are standard, Question: what is the nightstand made of ? 1. can't tell it's covered in cloth 2. it appears to be a large red pillow that may be leather 5. not sure 6. can't tell 7. some kind of metal , it's out of focus ...8. Wood 99. 0 100.I can't see a baggage cart MRR NDCG Figure 1: A visual dialog interaction. The question asks, “what is the nightstand made of ?”. We show our final ranking, created by the ensemble of an MRR/NDCG models’ rank- ings. The MRR/NDCG models are trained to optimize the MRR/NDCG metric. The MRR metric measures the number of retrievals to retrieve the human-derived answer. Hence, the MRR model favors human-like and detailed answers. On the other hand, the NDCG metric measures the rank of all the correct candidates based on dense annotation, which are often general and uncertain. Our ensemble approach seeks a mini- mal candidate set that …

Content truncated to 500 words