Even the Simplest Baseline Needs Careful Re-investigation: A Case Study on XML-CNN

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1987 - 2000 July 10-15, 2022 ©2022 Association for Computational Linguistics Even the Simplest Baseline Needs Careful Re-investigation: A Case Study on XML-CNN Si-An Chen1,2, Jie-Jyun Liu2, Tsung-Han Yang2, Hsuan-Tien Lin1, and Chih-Jen Lin1 1National Taiwan University {d09922007,htlin,cjlin}@csie.ntu.edu.tw 2ASUS Intelligent Cloud Services {eleven1_liu,henry1_yang}@asus.com Abstract The power and the potential of deep learning models attract many researchers to design ad- vanced and sophisticated architectures. Never- theless, the progress is sometimes unreal due to various possible reasons. In this work, through an astonishing example we argue that more ef- forts should be paid to ensure the progress in developing a new deep learning method. For a highly influential multi-label text classification method XML-CNN, we show that the supe- rior performance claimed in the original paper was mainly due to some unbelievable coinci- dences. We re-examine XML-CNN and make a re-implementation which reveals some con- tradictory findings to the claims in the original paper. Our study suggests suitable baselines for multi-label text classification tasks and con- firms that the progress on a new architecture cannot be confidently justified without a cau- tious investigation. 1 Introduction Deep learning has been a popular research topic in NLP due to its superior performance. The intrin- sic structure of deep learning allows researchers to enhance the model performance by introducing more complex network architectures. Neverthe- less, the increasing complexity brings difficulties to ensure the true architectural progress. For exam- ple, Adhikari et al. (2019) have shown that LSTM architectures with appropriate regularization are ei- ther competitive or superior to more recent models. As another example, Liu et al. (2021) report that the lack of hyperparameter tuning in an influential work (Mullenbach et al., 2018) makes the progress of subsequent network developments questionable. Complex architectures are more difficult to train, involve more hyperparameters, and are riskier to unintentional implementation. Because new archi- tectures are usually modified from previous ones, a questionable work may make the research progress unclear. Therefore, re-examining or reproducing influential architectures are now considered impor- tant in the community. In this work, we re-examine XML-CNN (Liu et al., 2017), an influential work in extreme multi- label text classification (XMTC), as a case study to demonstrate the demands of inspecting exist- ing architectures. XML-CNN has been viewed as an essential baseline in subsequent works (Peng et al., 2018; Prabhu et al., 2018; You et al., 2019; Chang et al., 2020; Adhikari et al., 2019) with more than hundreds of citations. XML-CNN roots from Kim-CNN (Kim, 2014), a classical architecture for multi-class text classification. The authors of XML-CNN proposed several modifications from Kim-CNN to accommodate the XMTC task and empirically claim that all modifications bring sig- nificant improvements. Despite XML-CNN’s popularity, we identified two serious implementation issues that make the original claims uncertain. First, the authors intro- duced dynamic max-pooling into XML-CNN, but the implementation is actually far from the intended formulation. Second, a bug in the …

Content truncated to 500 words