Label Smoothing for Text Mining

Proceedings of the 29th International Conference on Computational Linguistics, pages 2210–2219 October 12–17, 2022. 2210 Label Smoothing for Text Mining Peiyang Liu1,2,∗, Xiangyu Xi3,∗, Wei Ye1,† and Shikun Zhang1 1 National Engineering Research Center for Software Engineering, Peking University, Beijing, China, 2 PX Securities, Shenzhen, China, 3 Meituan, Beijing, China, {liupeiyang, wye, zhangsk}@pku.edu.cn, xixiangyu@meituan.com Abstract Current text mining models are trained with 0-1 hard label that indicates whether an instance belongs to a class, ignoring rich information of the relevance degree. Soft label, which in- volved each label of more varying degrees than the hard label, is considered more suitable for describing instances. The process of generating soft labels from hard labels is defined as label smoothing (LS). Classical LS methods focus on universal data mining tasks so that they ignore the valuable text features in text mining tasks. This paper presents a novel keyword-based LS method to automatically generate soft labels from hard labels via exploiting the relevance between labels and text instances. Generated soft labels are then incorporated into existing models as auxiliary targets during the training stage, capable of improving models without adding any extra parameters. Results of ex- tensive experiments on text classification and large-scale text retrieval datasets demonstrate that soft labels generated by our method con- tain rich knowledge of text features, improving the performance of corresponding models un- der both balanced and unbalanced settings. 1 Introduction Instances in most text mining datasets are usually assigned by one label. Such label, called hard label, reflects the logical relationship between the label and the instance. In most cases, hard labels are uti- lized for training models that learn to discriminate between classes with training objectives to maxi- mize the log probability of the correct class. How- ever, there are various real-world text mining tasks where there are not just two possibilities whether an instance belongs to a specific class since instances may be involved with multiple labels of varying degrees. In such scenarios, soft label (Bahri and Jiang, 2021; Hong et al., 2022), which involves ∗Co-first authors, contributed to this work equally †Corresponding author S1: It was nice. The cashier was chipper. Wish they wo- uld crank the A/C a little more though. Restroom was a little dirty. Hard Label (1, 0) Soft Label (0.6, 0.4) Table 1: An instance of sentiment classification with hard label and soft label. X denotes the label of being positive, while Y denotes negative in (X, Y). Positive and negative expressions are marked in red and blue, respectively. the explicit relative importance of each label, is a more reasonable description of an instance. For example, in the sentence S1 shown in the Table 1,1 though both positive and negative expressions exist in the sentence and sentiment slightly inclines to be positive, annotators must give a solid positive as a hard label. The soft label (0.6,0.4) describes the instance more comprehensively than hard labels. Few works are investigating soft labels for text data in the text mining community, although the issue …

Content truncated to 500 words