Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning
31219Authors
nan
Abstract
AbstractIn the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised multimodal correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (‘stay’, ‘step back’, or ‘step forward’). The proposed method reaches state-of-the-art performance on MSA.
Keywords
Full Content
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291 - 305 December 7-11, 2022 ©2022 Association for Computational Linguistics Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation Jannis Bulian Christian Buck Wojciech Gajewski Benjamin B¨orschinger Tal Schuster Google Research {jbulian, cbuck, wgaj, bboerschinger, talschuster}@google.com Abstract The predictions of question answering (QA) systems are typically evaluated against man- ually annotated finite sets of one or more an- swers. This leads to a coverage limitation that results in underestimating the true perfor- mance of systems, and is typically addressed by extending over exact match (EM) with pre- defined rules or with the token-level F1 mea- sure. In this paper, we present the first system- atic conceptual and data-driven analysis to ex- amine the shortcomings of token-level equiva- lence measures. To this end, we define the asymmetric no- tion of answer equivalence (AE), accepting an- swers that are equivalent to or improve over the reference, and publish over 23k human judgments for candidates produced by multi- ple QA systems on SQuAD.1 Through a care- ful analysis of this data, we reveal and quan- tify several concrete limitations of the F1 mea- sure, such as a false impression of graduality, or missing dependence on the question. Since collecting AE annotations for each eval- uated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE ap- proximations than F1, and to more accurately reflect the performance of systems. Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to ×2.6. 1 Introduction Automatically assessing the answers given by ques- tion answering (QA) systems for correctness can be nontrivial. This was already recognized in the early large-scale QA evaluation work of Voorhees 1Dataset and model information available at: https: //github.com/google-research-datasets/ answer-equivalence-dataset. The BEM model can be found at: https://tfhub.dev/ google/answer_equivalence/bem/1. and Tice (2000), leading them to suggest string- matching patterns for approximation, yet recogniz- ing their limitations: “it is quite difficult to determine auto- matically whether the difference between a new string and a judged string is sig- nificant with respect to the correctness of the answer.” Despite the early recognition of the importance and difficulty of evaluation for question answering, surprisingly little progress has been made. As of today, QA research in the NLP community relies almost exclusively on two token-level metrics, Ex- act Match (EM) and Token F1 (F1). Unfortunately, both fall short of capturing the difference between significant and insignificant span differences. In Section 2, we first document these limitations, pro- viding a detailed analysis and examples from the SQuAD dataset. Moreover, we extend this analysis with our new findings: the inherent asymmetry of the task and the reliance on the question, also pro- viding examples from the SQuAD dataset. We also identify cases where the context is important. One obvious …
Content truncated to 500 words
Labeled Text
AbstractIn the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised multimodal correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (‘stay’, ‘step back’, or ‘step forward’). The proposed method reaches state-of-the-art performance on MSA.
Publication Details
- Paper ID:
- 31219
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan