Querying Across Genres for Medical Claims in News

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 171–182, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 171 A Matter of Framing: The Impact of Linguistic Formalism on Probing Results Ilia Kuznetsov and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt http://www.ukp.tu-darmstadt.de/ Abstract Deep pre-trained contextualized encoders like BERT (Devlin et al., 2019) demonstrate re- markable performance on a range of down- stream tasks. A recent line of research in prob- ing investigates the linguistic knowledge im- plicitly learned by these models during pre- training. While most work in probing oper- ates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the for- malism used to annotate the underlying data. Can the choice of formalism affect probing re- sults? To investigate, we conduct an in-depth cross-formalism layer probing study in role se- mantics. We find linguistically meaningful dif- ferences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer prob- ing can detect subtle differences between the implementations of the same linguistic formal- ism. Our results suggest that linguistic formal- ism is an important dimension in probing stud- ies and should be investigated along with the commonly used cross-task and cross-lingual experimental settings. 1 Introduction The emergence of deep pre-trained contextualized encoders has had a major impact on the field of nat- ural language processing. Boosted by the availabil- ity of general-purpose frameworks like AllenNLP (Gardner et al., 2018) and Transformers (Wolf et al., 2019), pre-trained models like ELMO (Pe- ters et al., 2018) and BERT (Devlin et al., 2019) have caused a shift towards simple architectures where a strong pre-trained encoder is paired with a shallow downstream model, often outperforming the intricate task-specific architectures of the past. The versatility of pre-trained representations im- plies that they encode some aspects of general We asked John to leave . L=0 L=8 L=11 Figure 1: Intra-sentence similarity by layer L of the multilingual BERT-base. Functional tokens are similar in L = 0, syntactic groups emerge at higher levels. linguistic knowledge (Reif et al., 2019). Indeed, even an informal inspection of layer-wise intra- sentence similarities (Fig. 1) suggests that these models capture elements of linguistic structure, and those differ depending on the layer of the model. A grounded investigation of these regularities al- lows to interpret the model’s behaviour, design better pre-trained encoders and inform the down- stream model development. Such investigation is the main subject of probing, and recent studies con- firm that BERT implicitly captures many aspects of language use, lexical semantics and grammar (Rogers et al., 2020). Most probing studies use linguistics as a theoret- ical scaffolding and operate on a task level. How- ever, there often exist multiple ways to represent the same linguistic phenomenon: for example, En- glish dependency syntax can be encoded using a variety of formalisms, incl. Universal (Schuster …

Content truncated to 500 words