Learning Slice-Aware Representations with Mixture of Attentions

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1530–1536 August 1–6, 2021. ©2021 Association for Computational Linguistics 1530 Learning Slice-Aware Representations with Mixture of Attentions Cheng Wang Sungjin Lee Sunghyun Park Han Li Young-Bum Kim Ruhi Sarikaya Amazon Alexa AI {cwngam,sungjinl,sunghyu,lahl,youngbum,rsarikay}@amazon.com Abstract Real-world machine learning systems are achieving remarkable performance in terms of coarse-grained metrics like overall accuracy and F-1 score. However, model improvement and development often require fine-grained modeling on individual data subsets or slices, for instance, the data slices where the mod- els have unsatisfactory results. In practice, it gives tangible values for developing such mod- els that can pay extra attention to critical or interested slices while retaining the original overall performance. This work extends the re- cent slice-based learning (SBL) (Chen et al., 2019) with a mixture of attentions (MoA) to learn slice-aware dual attentive representa- tions. We empirically show that the MoA approach outperforms the baseline method as well as the original SBL approach on moni- tored slices with two natural language under- standing (NLU) tasks. 1 Introduction Though machine learning systems have been achieving excellent performance in terms of coarse- grained metrics like accuracy, they perform poorly or even fail on some individual data subsets (i.e., slices). For instance, many models have difficulties when learning for classes with only a few sam- ples or samples with challenging structures. In- specting particular data slices can serve as an im- portant component in model development cycles. A recently proposed slice-based learning (SBL) exhibited compelling results with more than 3% improvements on pre-defined slices (Chen et al., 2019) in the task of binary classification. However, one potential limitation of the existing attention mechanism in SBL is that in multi-class cases, the attention suffers from the difficulty in using the experts’ confidences appropriately for computing slice distributions (refer to Sec. 3). a1 a2 ak-1 ak r1 r2 rk-1 rk … … Backbone Model (slice expert) (shared head) (slice indicator) (membership attention) x s ∗ a1 a2 ak-1 ak x … ∗ r1 r2 rk-1 rk … × × (dot-product attention) (slice functions) 。 s1 s2 Figure 1: The slice-aware architecture with MoA. It consists of six components: (1) slice functions define the special data slices that we want to monitor; (2) backbone model for feature extraction (e.g., BERT); (3) slice indicators are membership functions to predict if a sample belongs to the slice; (4) slice experts aim to learn slice-specific representations; (5) shared head is the base task predictive layer across experts and (6) the proposed mixture of attentions (MoA) learns to attend to the slices of interest. It contains two different at- tention mechanism (red boxes): a membership atten- tion and a dot-product attention. The MoA learns to re-weight the expert representation r to a slice-aware representation s and the original representation x to s (yellow lines). The slice distributions are computed in deterministic (weighted sum of slices) or stochastic (sampling) way in re-weighting r and x. In this paper, we extend SBL with a …

Content truncated to 500 words