Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition Vered Shwartz Ido Dagan Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel vered1986@gmail.com dagan@cs.biu.ac.il Abstract Building meaningful phrase representations is challenging because phrase meanings are not simply the sum of their constituent meanings. Lexical composition can shift the meanings of the constituent words and introduce im- plicit information. We tested a broad range of textual representations for their capacity to address these issues. We found that, as ex- pected, contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information, in which their performance is still far from that of hu- mans. Our evaluation suite, consisting of six tasks related to lexical composition effects, can serve future research aiming to improve representations. 1 Introduction Modeling the meaning of phrases involves ad- dressing semantic phenomena that pose non- trivial challenges for common text representations, which derive a phrase representation from those of its constituent words. One such phenomenon is meaning shift, which happens when the meaning of the phrase departs from the meanings of its constituent words. This is especially common among verb-particle constructions (carry on), idi- omatic noun compounds (guilt trip), and other multi-word expressions (MWE, lexical units that form a distinct concept), making them ‘‘a pain in the neck’’ for NLP applications (Sag et al., 2002). A second phenomenon is common for both MWEs and free word combinations such as noun compounds and adjective-noun compositions. It happens when the composition introduces an implicit meaning that often requires world knowledge to uncover. For example, that hot refers to the temperature of tea but to the manner of debate (Hartung, 2015), or that olive oil is made of olives while baby oil is made for babies (Shwartz and Waterson, 2018). There has been a line of attempts to learn com- positional phrase representations (e.g., Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Wieting et al., 2017; Poliak et al., 2017), but many of these are tailored to a specific type of phrase or to a fixed number of constituent words, and they all disregard the surrounding context. Recently, contextualized word representations boasted dra- matic performance improvements on a range of NLP tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). Such models serve as a function for computing word representations in a given context, making them potentially more capable to address meaning shift. These models were shown to capture some world knowledge (e.g., Zellers et al., 2018), which may potentially help with uncovering implicit information. In this paper we test how well various text representations address these composition-related phenomena. Methodologically, we follow recent work that applied ‘‘black-box’’ testing to assess various capacities of distributed representations (e.g., Adi et al., 2017; Conneau et al., 2018). We construct an evaluation suite with six tasks related to the above two phenomena, as shown in Figure 1, and develop generic models that rely on pre-trained representations. We test six representations, including …

Content truncated to 500 words