QiaoNing at SemEval-2020 Task 4: Commonsense Validation and Explanation System Based on Ensemble of Language Model
24609Authors
nan
Abstract
AbstractThe ability of common sense validation and explanation is very important for most models. Most obviously, this will directly affect the rationality of the generated model output. The large amount and diversity of common sense poses great challenges to this task. In addition, many common sense expressions are obscure, thus we need to understand the meaning contained in the vocabulary in order to judge correctly, which further increases the model’s requirements for the accuracy of word representation. The current neural network models are often data-driven, while the annotated data is often limited and requires a lot of manual labeling. In such case, we proposed transfer learning to handle this challenge. From our experiments, we can draw the following three main conclusions: a) Neural language model fully qualified for commonsense validation and explanation. We attribute this to the powerful word and sentence representation capabilities of language models. b) The inconsistency of task of pre-training and fine-tuning will badly hurt the performance. c) A larger amount of corpus and more parameters will enhance the common sense of the model. At the same time, the content of the corpus is equally important.
Keywords
Full Content
Proceedings of the 14th International Workshop on Semantic Evaluation, pages 415–421 Barcelona, Spain (Online), December 12, 2020. 415 QiaoNing at SemEval-2020 Task 4: Commonsense Validation and Explanation system based on ensemble of language model Pai Liu Northeastern University, Shenyang, China pai.liu.1998@gmail.com Abstract In this paper, we present language model system submitted to SemEval-2020 Task 4 competition: ”Commonsense Validation and Explanation”. We participate in two subtasks for subtask A: validation and subtask B: Explanation. We implemented with transfer learning using pretrained language models (BERT, XLNet, RoBERTa, and ALBERT) and fine-tune them on this task. Then we compared their characteristics in this task to help future researchers understand and use these models more properly. The ensembled model better solves this problem, making the model’s accuracy reached 95.9% on subtask A, which just worse than human’s by only 3% accuracy. 1 Introduction This task (Wang et al., 2019)is to evaluate how well a model can do for sense making on English data set by its ability to judge whether natural language sentences are not in line with common sense and explain the reasons. To thoroughly evaluate the model, three subtasks are designed in (Wang et al., 2020): a) Subtask A Validation - Given two statements with similar structures, the task is to discern which statement relatively makes sense and which one does not. b) Subtask B Explanation(Multiple-choices) - From the three candidate reasons given, choose the reason that is most likely to explain a statement that doesn’t make sense. c) Subtask C Explanation(Generation) - Generating explanations for non-common sense statements. Natural Language Understanding (NLU) has drawn an increasing amount of research attention recently. Although some well-performed end-to-end model even show better performances than humans on some benchmarks, they are still far from human on common sense. If a model lack of common sense processes data that requires common sense, the performance of the model may drop dramatically, which is a sign for poor robustness. However, building models that are robust to adversarial attacks is essential(Yang et al., 2020). Therefore, the importance of commonsense reasoning is not only confined to NLU system, it should also be expanded to other systems. Many similar tasks have been studied and solved before this task was proposed. Previous attempts on solving common sense challenge usually involve heavy utilization of annotated knowledge bases(Peng et al., 2015), rule-based reasoning(Bailey et al., 2015), or hand-crafted features(Sch¨uller, 2014). Over time, the advantages of the knowledge base gradually emerged from other methods, and researchers are more inclined to use this method to solve such problems. Simple models based on distributional representations perform poorly on this task, despite doing well on selection preference, but injecting manually elicited knowledge about entity properties provides a substantial performance boost(Wang et al., 2018). However, using annotated knowledge bases is a expensive approach while unsupervised training is done on text corpora which can be cheaply curated. Nowadays, neural LMs having achieved great success being used as feature representations for a sentence, or a paragraph, which improves NLP applications in …
Content truncated to 500 words
Labeled Text
AbstractThe ability of common sense validation and explanation is very important for most models. Most obviously, this will directly affect the rationality of the generated model output. The large amount and diversity of common sense poses great challenges to this task. In addition, many common sense expressions are obscure, thus we need to understand the meaning contained in the vocabulary in order to judge correctly, which further increases the model’s requirements for the accuracy of word representation. The current neural network models are often data-driven, while the annotated data is often limited and requires a lot of manual labeling. In such case, we proposed transfer learning to handle this challenge. From our experiments, we can draw the following three main conclusions: a) Neural language model fully qualified for commonsense validation and explanation. We attribute this to the powerful word and sentence representation capabilities of language models. b) The inconsistency of task of pre-training and fine-tuning will badly hurt the performance. c) A larger amount of corpus and more parameters will enhance the common sense of the model. At the same time, the content of the corpus is equally important.
Publication Details
- Paper ID:
- 24609
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan