SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1039 - 1049 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts Michael Roth Talita Anthonio Anna Sauer University of Stuttgart Institute for Natural Language Processing {michael.roth,talita.anthonio,anna.sauer}@ims.uni-stuttgart.de Abstract We describe SemEval-2022 Task 7, a shared task on rating the plausibility of clarifications in English-language instructional texts. The dataset for this task consists of manually clar- ified how-to guides for which we generated alternative clarifications and collected human plausibility judgements.1 The task of partici- pating systems was to automatically determine the plausibility of a clarification in the respec- tive context. In total, 21 participants took part in this task, with the best system achieving an accuracy of 68.9%. This report summarizes the results and findings from 8 teams and their system descriptions. Finally, we show in an ad- ditional evaluation that predictions by the top participating team make it possible to identify contexts with multiple plausible clarifications with an accuracy of 75.2%. 1 Introduction Understanding texts in natural language requires that both explicit text components as well as im- plicit references and relationships are interpreted correctly. This applies in particular to instructional texts, which demand a clear understanding of in- dividual instruction steps in order to reach the de- sired goal. Possible uncertainties should therefore already be clarified in the text. In principle, such clarifications can also be generated automatically. In that case, however, it will be necessary to inves- tigate the circumstances under which a clarification is plausible and unambiguous. As a first step towards such an investigation, this shared task evaluates the ability of NLP systems to distinguish between plausible and implausible clar- ifications of an instruction. Inspired by the success of previous cloze-based evaluations (see Section 2), we set up our task as a cloze task, in which clarifi- cations are presented as fillers and systems have to 1The task data is available at https://github.com/ acidAnn/claire. Choose a Hair Salon (1) Check ratings of different salons. (2) Visit the salon’s website. (3) Call ∅ and ask questions. ✓ the salon ✗ a friend ✓ the number ✗ your stylist ✓ the owner Table 1: Simplified example from a pilot study: the top shows a sentence (3) and shortened version of its dis- course context (1–2). In the clarified version of this sen- tence, the phrase the salon was inserted. Other phrases shown in the bottom part are automatically generated fillers, annotated as plausible (✓) or implausible (✗). identify which fillers plausibly fit in a given context (see Table 1). Our focus in this task is on differ- ent types of referring expressions that are either underspecified or not realized explicitly at all, and we consider possible clarifications in the form of additional specification or explicitation. Research in linguistics and psychology has shown that individuals use language differently (Pennebaker and King, 1999; Heylighen and De- waele, 2002). In particular when it comes to im- plicit …

Content truncated to 500 words