SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 228 - 238 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 3: PreTENS – Evaluating Neural Networks on Presuppositional Semantic Knowledge Roberto Zamparelli1, Shammur A Chowdhury2, Dominique Brunato3, Cristiano Chesi4, Felice Dell’Orletta3, Arid Hasan5, Giulia Venturi3 1CIMeC, University of Trento, Rovereto Italy; 2Qatar Computing Research Institute, HBKU, Qatar; 3ILC-CNR, Pisa, Italy; 4NETS-IUSS, Pavia, Italy; 5Daffodil International University, Dhaka, Bangladesh roberto.zamparelli@unitn.it, shchowdhury@hbku.edu.qa, cristiano.chesi@iusspavia.it, arid.cse0325.c@diu.edu.bd, [name.surname]@ilc.cnr.it Abstract We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continu- ous scale. The sentences were artificially gen- erated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on var- ious types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systems reached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in spe- cific constructions and/or languages. 1 Introduction A growing body of literature on the cognitive side of computational linguistics has tried to probe the ability of language models to recognize deviant linguistic structures. Recognizing linguistic ill- formedness requires some degree of metalinguis- tic awareness in adult humans (i.e. the ability to say not just that there is ‘something wrong’ in a sentence, but also where the problem lies or how the sentence could be improved), and it is not clear whether and to what extent artificial systems can induce this type of knowledge purely from exposure to raw linguistic data (Linzen and Ba- roni, 2020). Most of the previous investigations on this topic have focused on phenomena that are purely syntactic (agreement, Gulordava et al. 2018; dislocated arguments with island effects, Wilcox et al. (2018); Warstadt et al. (2019); Chowdhury and Zamparelli (2018), clause embedding, Futrell et al. 2019, etc.) or at the syntax/semantics inter- face (negative-polarity items, Jumelet and Hupkes 2018; argument structure, quantifier restrictions, Warstadt et al. 2019), mostly using LSTM archi- tectures (but see Tran et al. 2018; Ettinger 2020). The fact that many purely semantic effects result in the (non) availability of certain readings (e.g. the scope of a quantifier over a higher negation, in “I didn’t see some people”) makes it of course harder to translated them into computationally testable paradigms. The task we describe in this paper focuses on an area of purely semantic competence that, to the best of our knowledge, is unexplored in the com- putational literature, and one which gives rise to a robust intuition of deviance, triggered by the failure of a type of presupposition: the requirement for two nominal arguments to be (or not be) in an (or- dered) taxonomic relation. These presuppositions are introduced by a …

Content truncated to 500 words