ISTI-CNR at SemEval-2016 Task 4: Quantification on an Ordinal Scale

Proceedings of SemEval-2016, pages 92–95, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics ISTI-CNR at SemEval-2016 Task 4: Quantification on an Ordinal Scale Andrea Esuli Istituto di Scienza e Tecnologie dell’Informazione “A. Faedo” via G. Moruzzi, 1 56124, Pisa, ITALY andrea.esuli@isti.cnr.it Abstract This paper details on the participation of ISTI- CNR to task 4 of Semeval 2016. Among the five subtasks, special attention has been paid to the five-point scale quantification sub- task. The quantification method we pro- pose is based on the observation that a standard document-by-document regression method usually has a bias towards assigning high prevalence labels. Our method models such bias with a linear model, in order to com- pensate it and to produce the quantification es- timates. 1 Introduction The participation of ISTI-CNR to task 4 of Semeval 2016 (Nakov et al., 2016) produced submissions for all the five proposed subtasks. Submissions for subtasks A and B are based on a relatively typical machine learning pipeline, with B used as the base classification tool for sub- task D, which uses a quantification via classifica- tion method. Subtask C uses an ordinal regression method, based on building a data-balanced tree of binary classifiers. The regression method of sub- task C has been used as the base regression tool for the implementation of the quantification method for subtask E. We propose a novel quantification method for subtask E, tweet quantification according to a five- point scale. The method stems from the intuition of measuring and compensating the bias a regres- sion model may have for the labels with high preva- lences. All the code we produced for these tasks is mainly based on the scikit-learn python library (Pedregosa et al., 2011) and it is published under an open source license1. The next sections detail on the data and methods adopted to produce the five submissions. 2 Training data The labeled training dataset has been downloaded using the tool suggested by the organizers2. Table 1 summarizes the total number of tweets available for download at the time of crawling (November 2015). The final number of tweets used to train the classi- fiers, or quantifiers, is 6223 for subtasks A, C and E, and 4475 for subtasks B and D. A small number of tweets in the dataset appeared multiple times, some with conflicting labels. In the “train” data parts, for example, 25 tweets appeared twice for subtasks A, C, and E, six of them with conflicting labels for subtask A, and 14 with con- flicting labels for subtasks C and E3. For subtasks B, D, the number of tweets appearing twice in train- ing is 13, one of them with conflicting labels. Du- plicate tweets have been reduced to a single instance and those with conflicting labels have been excluded from the dataset and from any analysis performed in this work. 1https://github.com/aesuli/ semeval2016-task4 2https://github.com/aritter/twitter_ download 3Subtask A uses a coarse-grained three-point scale, so it may happen that a conflicting ‘-1’/‘-2’ labeling of a tweet for sub- …

Content truncated to 500 words