Mxgra at SemEval-2020 Task 4: Common Sense Making with Next Token Prediction

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 569–573 Barcelona, Spain (Online), December 12, 2020. 569 Team Mxgra at SemEval-2020 Task 4: common sense making with next token prediction Kris Collins kristopher.collins @student. uni-tuebingen.de Max Grathwohl max-peter.grathwohl @student. uni-tuebingen.de Heba Ahmed hebah.ahmed @student. uni-tuebingen.de Abstract In this paper, we explore solutions to a common sense making task in which a model must discern which of two sentences is against common sense. We used a pre-trained language model which we used to calculate perplexity scores for input to discern which sentence contained an unlikely sequence of tokens. Other approaches we tested were word vector distances, which were used to find semantic outliers within a sentence, and siamese network. By using the pre-trained language model to calculate perplexity scores based on the sequence of tokens in input sentences, we achieved an accuracy of 75 percent. 1 Introduction Much research has been conducted on Natural Language Understanding (NLU) and models based on neural networks have achieved state-of-the-art results in reading comprehension . However, results in tasks regarding common sense making do not rival those of human performance, such as in inference tasks in which state-of-the-art models score upwards of 80 percent (Devlin et al., 2019). As the organizers of the SemEval 2020 Commonsense Validation and Explanation (Wang et al., 2020) task highlighted, pinpointing exactly why an utterance is against common sense is not a trivial undertak- ing. Advances in this area could lead to improvements in NLU. We participated only in Task A, which was determining which of two sentences was against common sense. For example, given the input “He poured milk on his cereal” and “He poured orange juice on his cereal” we would identify “He poured orange juice on his cereal” as the sentence against common sense. Task A was evaluated by accuracy. All data was in English. We used a pre-trained language model to calculate the probability of the next word in a sequence to produce perplexity scores for each of the sentences in the input pairs. A higher perplexity score meant a lower overall probability of the words occurring in this order, thus being more likely against commonsense. We also attempted to solve this task with word vector distances, attempting to find an outlier among tokens in each input sentence, and with a siamese network, which was trained as a feature embedder for valid sentences. 2 Background The data consisted of pairs of English sentences, one of which made sense and the other did not. The desired output was the index of the sentence against common sense. For example, the model would take the input sentences “He poured orange juice on his cereal.” and “He poured milk on his cereal.” and output a 0. The official training data from the ComEV organizers was a collection of 10,000 pairs of sentences. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 570 2.1 Pre-Trained Models Though there are applications that intend to …

Content truncated to 500 words