Layer Selection Strategy

We generalize deep self-attention distillation in MINILM (Wang et al., 2020) by only us- ing self-attention relation distillation for task- agnostic compression of pretrained Transform- ers. In particular, we define …...