PIC a Different Word: A Simple Model for Lexical Substitution in Context

Proceedings of NAACL-HLT 2016, pages 1121–1126, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics PIC a Different Word: A Simple Model for Lexical Substitution in Context Stephen Roller Department of Computer Science The University of Texas at Austin roller@cs.utexas.edu Katrin Erk Department of Linguistics The University of Texas at Austin katrin.erk@mail.utexas.edu Abstract The Lexical Substitution task involves select- ing and ranking lexical paraphrases for a target word in a given sentential context. We present PIC, a simple measure for estimating the ap- propriateness of substitutes in a given con- text. PIC outperforms another simple, com- parable model proposed in recent work, es- pecially when selecting substitutes from the entire vocabulary. Analysis shows that PIC improves over baselines by incorporating fre- quency biases into predictions. 1 Introduction Lexical substitution (McCarthy and Navigli, 2009) is a task in which word meaning in context is de- scribed not through dictionary senses but through substitutes (paraphrases) chosen by annotators. For example, consider the following usage of the adjec- tive bright: “The bright girl was reading a book.” Valid lexical substitutions for bright include adjec- tives like smart and intelligent, but not words like luminous or colorful. Originally introduced as a SemEval task in 2007, lexical substitution has often been used to evaluate the ability of distributional models to handle pol- ysemy (Erk and Pad´o, 2008; Thater et al., 2010; Dinu and Lapata, 2010; Van de Cruys et al., 2011; Melamud et al., 2015b; Melamud et al., 2015a; Kawakami and Dyer, 2015). Recent models include a simple but high-performing method by Melamud et al. (2015b), which uses the Skip-gram model of Mikolov et al. (Mikolov et al., 2013) to compute the probability of a substitute given a sentence context, and integrates it with the probability of the substi- tute given the target. The current state of the art is held by another model of Melamud (Melamud et al., 2015a), which uses a more complex architecture. In this paper we build on the simple model of Melamud et al. (2015b), as simpler methods are eas- ier to recreate and integrate into larger pipelines.1 We explore a weak form of supervision that recently has proved beneficial on many NLP tasks: using a language modeling task on unannotated data. We find a strong improvement over Melamud’s simple measure, particularly on the all-words ranking task. Interestingly, analysis of PIC shows it improves over baselines by incorporating frequency biases into pre- dictions. 2 Prior Work In the lexical substitution task, an annotator is given a target word in context and generates one or more substitutes. As multiple annotators label a target, the result is a weighted list of substitutes, where weights indicate how many annotators chose a par- ticular substitute (McCarthy and Navigli, 2009). There have been numerous approaches on the lex- ical substitution task of varying complexity and us- ing various lexical resources (McCarthy and Nav- igli, 2007). Some approaches focus on explic- itly modeling an in-context vector (Erk and Pad´o, 2008; Dinu and Lapata, 2010; …

Content truncated to 500 words