Building Large-Scale English and Korean Datasets for Aspect-Level Sentiment Analysis in Automotive Domain

Proceedings of the 28th International Conference on Computational Linguistics, pages 961–966 Barcelona, Spain (Online), December 8-13, 2020 961 Building Large-Scale English and Korean Datasets for Aspect-Level Sentiment Analysis in Automotive Domain Dongmin Hyun, Junsu Cho, Hwanjo Yu∗ Dept. of Computer Science and Engineering, POSTECH, Pohang, Republic of Korea {dm.hyun, junsu7463, hwanjoyu}@postech.ac.kr Abstract We release large-scale datasets of users’ comments in two languages, English and Korean, for aspect-level sentiment analysis in automotive domain. The datasets consist of 58,000+ comment- aspect pairs, which are the largest compared to existing datasets. In addition, this work covers new language (i.e., Korean) along with English for aspect-level sentiment analysis. We build the datasets from automotive domain to enable users (e.g., marketers in automotive companies) to analyze the voice of customers on automobiles. We also provide baseline performances for future work by evaluating recent models on the released datasets. 1 Introduction Aspect-level sentiment analysis (ALSA) has been actively studied to understand authors’ opinion on aspects from texts. For example, in a given text, “Although the space is smaller than most, it is the best service you will find in even the largest restaurants”, the author’s sentiment to space and service is negative and positive, respectively. Since devising deep learning models for ALSA recently received substantial attention (Zeng et al., 2019), building large-scale datasets in different languages has been an essential line of research (Rosenthal et al., 2017). However, due to the high cost of human annotation, the size and language of datasets are still limited. Specifically, only one of public datasets contains more than 20,000 instances, and existing datasets cover only three languages (i.e., English, Spanish, and Arabic). To this end, we release large-scale datasets of users’ comments with two languages such as English and Korean in automotive domain. The total size of these datasets is 58,603, which is the largest compared to existing datasets for ALSA. In addition, the datasets include new language (i.e., Korean), which extends the coverage of the datasets in terms of languages. In this work, we focus on automotive domain to build the datasets to enable users (e.g., marketers in automotive companies) to analyze the voice of customers on aspects related to automobiles. To build the datasets, domain experts define the 12 largest automotive manufacturers (e.g., Ford) by production volume and their popular automobiles (e.g., Mustang) as aspects. Given the aspects, we collect users’ comments from automotive communities in the United States and South Korea. To annotate aspect-level sentiments, we perform crowdsourcing by assigning at least three annotators to each comment-aspect pair. The annotated datasets consist of 28,571 and 30,032 comment-aspect pairs in English and Korean, respectively. Inter-annotator agreements are 0.36 (fair agreement) for English dataset and 0.54 (fair agreement) for Korean dataset in terms of Fleiss’ kappa1. We perform extensive experiments with deep learning models for ALSA to provide the baseline per- formance on the released datasets for future work. The datasets are publicly available at our website. 2 Related Datasets Researchers in ALSA have built labeled datasets as supervised …

Content truncated to 500 words