More than meets the eye: Study of Human Cognition in Sense Annotation

Proceedings of NAACL-HLT 2013, pages 733–738, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics More than meets the eye: Study of Human Cognition in Sense Annotation Salil Joshi IBM Research India Bangalore, India saljoshi@in.ibm.com Diptesh Kanojia Gautam Buddha Technical University Lucknow, India dipteshkanojia@gmail.com Pushpak Bhattacharyya Computer Science and Engineering Department Indian Institute of Technology, Bombay Mumbai, India pb@cse.iitb.ac.in Abstract Word Sense Disambiguation (WSD) ap- proaches have reported good accuracies in recent years. However, these approaches can be classified as weak AI systems. According to the classical definition, a strong AI based WSD system should perform the task of sense disambiguation in the same manner and with similar accuracy as human beings. In order to accomplish this, a detailed understanding of the human techniques employed for sense disambiguation is necessary. Instead of building yet another WSD system that uses contextual evidence for sense disambiguation, as has been done before, we have taken a step back - we have endeavored to discover the cognitive faculties that lie at the very core of the human sense disambiguation technique. In this paper, we present a hypothesis regard- ing the cognitive sub-processes involved in the task of WSD. We support our hypothesis using the experiments conducted through the means of an eye-tracking device. We also strive to find the levels of difficulties in annotating vari- ous classes of words, with senses. We believe, once such an in-depth analysis is performed, numerous insights can be gained to develop a robust WSD system that conforms to the prin- ciple of strong AI. 1 Introduction Word Sense Disambiguation (WSD) is formally defined as the task of computationally identifying senses of a word in a context. The phrase ‘in a context’ is not defined explicitly in the literature. NLP researchers define it according to their conve- nience. In our current work, we strive to unravel the appropriate meaning of contextual evidence used for the human annotation process. Chatterjee et al. (2012) showed that the contextual evidence is the predominant parameter for the human sense annotation process. They also state that WSD is successful as a weak AI system, and further analysis into human cognitive activities lying at the heart of sense annotation can aid in development of a WSD system built upon the principles of strong AI. Knowledge based approaches, which can be con- sidered to be closest form of WSD conforming to the principles of strong AI, typically achieve low accuracy. Recent developments in domain-specific knowledge based approaches have reported higher accuracies. A domain-specific approach due to Agirre et al. (2009) beats supervised WSD done in generic domains. Ponzetto and Navigli (2010) present a knowledge based approach which rivals the supervised approaches by using the semantic relations automatically extracted from Wikipedia. They reported approximately 7% gain over the closet supervised approach. In this paper, we delve deep into the cognitive roles associated with sense disambiguation through the means of an eye-tracking device capturing the gaze patterns of lexicographers, during the annotation process. In-depth discussions with trained …

Content truncated to 500 words