AutoTriggER: Label-Efficient and Robust Named Entity Recognition with Auxiliary Trigger Extraction

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 274–285 May 2-6, 2023 ©2023 Association for Computational Linguistics Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models Haotian Xu Ailbaba Group htxu91@gmail.com Yingying Zhang East China Normal University yyzhang@fem.ecnu.edu.cn Abstract Though pre-trained language models achieve notable success in many applications, it’s usu- ally controversial for over-confident predic- tions. Specifically, the in-distribution (ID) mis- calibration and out-of-distribution (OOD) de- tection are main concerns. Recently, some works based on energy-based models (EBM) have shown great improvements on both ID calibration and OOD detection for images. However, it’s rarely explored in natural lan- guage understanding tasks due to the non- differentiability of text data which makes it more difficult for EBM training. In this pa- per, we first propose a triple-hybrid EBM which combines the benefits of classifier, con- ditional generative model and marginal genera- tive model altogether. Furthermore, we lever- age contrastive learning to approximately train the proposed model, which circumvents the non-differentiability issue of text data. Exten- sive experiments have been done on GLUE and six other multiclass datasets in various domains. Our model outperforms previous methods in terms of ID calibration and OOD detection by a large margin while maintaining competitive accuracy. 1 Introduction Since many industrial applications involve safety -critical domains such as healthcare (Li et al., 2019; Blinov et al., 2020; Li et al., 2020; Rasmy et al., 2021; Sarabadani, 2019), anticipating credit card defaults (Sun and Vasarhalyi, 2021) and self- driving (Khaitan et al., 2021), it’s essential for ma- chine learning systems to provide not only accurate but also well-calibrated predictions (Li et al., 2019), which can help to decide whether it can be trusted. However, models achieving high accuracy usu- ally lead to overconfidence and miscalibration (Guo et al., 2017; Thulasidasan et al., 2019; Ovadia et al., 2019). This motivates an interesting and important area that attempts to achieve a better trade-off be- tween accuracy and calibration. In addition to ID calibration, it’s more important for machine learn- ing models to produce high uncertainty when OOD data is observed, rather than to produce wrong yet wildly confident predictions. Related works. To overcome the problem of mis- calibration, numerous methods have been proposed. The natural way is post-hoc calibration that trans- forms the output of the original network into cal- ibrated confidence scores while maintaining the network’s accuracy (Guo et al., 2017; Rahimi et al., 2020; Jung et al., 2020). The second method to mit- igate miscalibration is to add regularizations dur- ing training such as label smoothing (Wang et al., 2020), Mixup (Zhang et al., 2018). Desai and Dur- rett (2020) and Kong et al. (2020) further conveys that the aforementioned methods can be applied to improve the calibration of pre-trained language models on NLU tasks. The third way is to design a specific loss function to minimize the discrepancy between accuracy and confidence. For example, Kong et al. (2020) lately propose the ID and OOD regularizer to …

Content truncated to 500 words