Stanford MLab at SemEval 2022 Task 7: Tree- and Transformer-Based Methods for Clarification Plausibility

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1067 - 1070 July 14-15, 2022 ©2022 Association for Computational Linguistics Stanford MLab at SemEval 2022 Task 7: Tree- and Transformer-Based Methods for Clarification Plausibility Thomas Yim∗, Junha Lee∗, Rishi Verma, Scott Hickmann, Annie Zhu, Camron Sallade, Ian Ng†, Ryan Chi†, and Patrick Liu† Stanford University {yimt, junhalee, rishirv, hickmann, anniezhu}@stanford.edu, {camron, iyhn8192, ryanchi, pliu1}@stanford.edu Abstract In this paper, we detail the methods we used to determine the idiomaticity and plausibility of candidate words or phrases into an instructional text as part of the SemEval Task 7: Identifying Plausible Clarifications of Implicit and Under- specified Phrases in Instructional Texts. Given a set of steps in an instructional text, there are certain phrases that most plausibly fill that spot. We explored various possible architectures, in- cluding tree-based methods over GloVe embed- dings, ensembled BERT and ELECTRA mod- els, and GPT 2-based infilling methods. 1 Introduction The internet is filled with instructional texts from websites like wikiHow that detail how to perform a variety of tasks (from tying a bow tie to build- ing a deck of stairs). With this increase of quan- tity and use of instructional texts, it has become increasingly important for them to be clear and unambiguously worded. To this end, we evaluate whether lightweight and neural models are capable of detecting which phrases most plausibly fit into a given series of instructions. The current revision process requires that a reader potentially identify something wrong about the content, and they report it to the website for al- terations. A system such as the one described could automatically identify candidates that are unlikely to exist and report them for human verification. In this paper, we describe our lightweight and transformer-based models that rank the plausibility of candidate phrases given some previous context. 2 Background 2.1 Task Setup The data for this task was provided by SemEval Task 7 (Roth et al., 2022). This dataset is an aug- mented version of WikiHowToImprove (Anthonio ∗Co-first authors. †Co-senior authors. et al., 2020), which consists of 2.7 million sen- tences and their revision histories extracted from the instructional website WikiHow. The dataset extracts over 4000 sentences and revision that rep- resent clarifications of the original text. Each sen- tence is masked and presented with 5 possible fillers that may represent a clarification. The article title, subsection, previous context, and future con- text are also provided. Each filler is annotated with a plausibility class label (either IMPLAUSIBLE, NEUTRAL, or PLAUSIBLE), whose prediction is the basis of Subtask A, and a plausibility score on a scale from 1 to 5, whose prediction is the basis of Subtask B. The plausibility of a given filler is highly depen- dent on the context. For example, the clarification birthday is annotated as implausible for the sen- tence 2. Send a card. Even if you’re able to have the conversation in person, it’s worth sending a card. given the previous context 1. Ask in person whenever possible. Receiving …

Content truncated to 500 words