Graded and Word-Sense-Disambiguation Decisions in Corpus Pattern Analysis: a Pilot Study

Graded and Word-Sense-Disambiguation Decisions in Corpus Pattern Analysis: a Pilot Study ♣Silvie Cinkov´a, ♣Ema Krejˇcov´a, ♣Anna Vernerov´a, ♠V´ıt Baisa ♣Charles University in Prague, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics ♠Masaryk University, Brno, Faculty of Informatics, NLP Centre {cinkova,krejcova,vernerova}@ufal.mff.cuni.cz, xbaisa@fi.muni.cz Abstract We present a pilot analysis of a new linguistic resource, VPS-GradeUp (available at http://hdl.handle.net/11234/1-1585). The resource contains 11,400 graded human decisions on usage patterns of 29 English lexical verbs, randomly selected from the Pattern Dictionary of English Verbs (Hanks, 2000 2014). The selection was random and based on their frequency and the number of senses their lemmas have in PDEV. This data set has been created to observe the interannotator agreement on PDEV patterns produced using the Corpus Pattern Analysis (Hanks, 2013). Apart from the graded decisions, the data set also contains traditional Word-Sense-Disambiguation (WSD) labels. We analyze the associations between the graded annotation and WSD annotation. The results of the respective annotations do not correlate with the size of the usage pattern inventory for the respective verbs lemmas, which makes the data set worth further linguistic analysis. Keywords: CPA, graded decisions, English, verbs, usage patterns, annotation, Likert scales 1. Introduction We investigate human graded decisions about the goodness of match between lexicographically defined usage patterns of verbs and random corpus concordances of those verbs in terms of interannotator agreement. The usage patterns orig- inate from PDEV – the Pattern Dictionary of English Verbs (Hanks, 2000 2014) –, and the concordances come from the BNC (British National Corpus Consortium, 2007). PDEV is a lexical resource driven by the Corpus Pattern Analysis – an interesting method of lexical description (Hanks, 2013) based on manual syntactico-semantic clustering of random concordances of a verb into usage patterns. Lexical entries produced by this method are intuitively very appealing for human language learners. Besides, a number of attempts have been pursued to introduce PDEV as a resource or Corpus Pattern Analysis as an approach into the computational linguistics – for instance, several recent SemEval tasks targeted CPA (Baisa et al., 2015). While the computational tasks encompass classification of unseen sentences according to PDEV patterns already available as well as generating CPA-style lexical entries from large data, our investigation aims at what factors play a role in the human classification and clustering decisions. We hope that this research will help with a more nuanced evaluation of the classification and entry-building tasks. As a first step, we built a toy lexical resource with 29 PDEV entries and 50-item batches of the corresponding concordances. It contains two types of annotation: • graded decisions about the goodness of match between a concordance and each pattern listed in the corre- sponding PDEV entry; • traditional WSD decisions (the best matching pattern for each concordance). This lexical resource is called VPS-GradeUp ((Baisa et al., 2015) and is available through the LINDAT-CLARIN repository at http://hdl.handle.net/11234/1-1585). It contains 11,400 graded human decisions as well as 1450 WSD decisions concerning these 29 verbs: seal, sail, distinguish, adjust, cancel, need, …

Content truncated to 500 words