Learning Features from Co-occurrences: A Theoretical Analysis

Proceedings of the 27th International Conference on Computational Linguistics, pages 2846–2854 Santa Fe, New Mexico, USA, August 20-26, 2018. 2846 Learning Features from Co-occurrences: A Theoretical Analysis Yanpeng Li IBM T. J. Watson Research Center Yorktown Heights, New York 10598, USA liyanpeng.lyp@gmail.com Abstract Representing a word by its co-occurrences with other words in context is an effective way to capture the meaning of the word. However, the theory behind remains a challenge. In this work, taking the example of a word classification task, we give a theoretical analysis of the approaches that represent a word X by a function f(P(C|X)), where C is a context feature, P(C|X) is the conditional probability estimated from a text corpus, and the function f maps the co-occurrence measure to a prediction score. We investigate the impact of context feature C and the function f. We also explain the reasons why using the co-occurrences with multiple context features may be better than just using a single one. In addition, based on the analysis, we propose a hypothesis about the conditional probability on zero probability events. 1 Introduction In natural language processing (NLP) and information retrieval (IR), representing a word by its co- occurrences with contexts is an effective way to learn the meaning of the word and lead to significant improvement in many tasks (Deerwester et al., 1990; Brown et al., 1992; Mikolov et al., 2013). The underlying intuition known as distributional hypothesis (Harris, 1954) can be summarized as: "a word is characterized by the company it keeps" (Firth, 1957). However, there is a lack of theory to justify why it works, even for a simple task such as word classification or clustering. For example, to determine if the word “phycoerythrin” is a gene name, we found it was effective to use the ratio of the count of “phycoerythrin gene” to the count of “phycoerythrin” as features (Li et al., 2009), denoted by ˆP(“Xgene”|X = phycoerythrin) = Count(phycoerythrin gene) Count(phycoerythrin) The ratio is equivalent to the estimation of the conditional probability P(“Xgene”|X = phycoerythrin) from a large text corpus. The pattern “Xgene” is a context feature that indicates if the word “gene” appears right next to the word X. Assuming Y ∈ {0, 1} is the gold standard label such as “being a gene name” and C ∈ {0, 1} is a binary context feature such as “Xgene”. The nature of the approach is to predict P(Y = 1|X) by P(C = 1|X) based on the intuition that there could be correlation between these two functions of X. Also P(Y = 1|X) tends to be more difficult to estimate than P(C = 1|X), since the gold standard labels are more expensive to obtain than the context patterns. We need to know why and when P(C = 1|X) is effective to predict P(Y = 1|X). Obviously, this study is beyond computational linguistics only but of general interest of probability and statistical theory. In this work, we first show that in a word classification task, simple co-occurrence …

Content truncated to 500 words