ExpBERT: Representation Engineering with Natural Language Explanations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2106–2113 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2106 ExpBERT: Representation Engineering with Natural Language Explanations Shikhar Murty Pang Wei Koh Percy Liang Computer Science Department, Stanford University {smurty,pangwei,pliang}@cs.stanford.edu Abstract Suppose we want to specify the inductive bias that married couples typically go on hon- eymoons for the task of extracting pairs of spouses from text. In this paper, we allow model developers to specify these types of inductive biases as natural language explana- tions. We use BERT fine-tuned on MultiNLI to “interpret” these explanations with respect to the input sentence, producing explanation- guided representations of the input. Across three relation extraction tasks, our method, ExpBERT, matches a BERT baseline but with 3–20× less labeled data and improves on the baseline by 3–10 F1 points with the same amount of labeled data. 1 Introduction Consider the relation extraction task of finding spouses in text, and suppose we wanted to specify the inductive bias that married couples typically go on honeymoons. In a traditional feature en- gineering approach, we might try to construct a “did they go on a honeymoon?” feature and add that to the model. In a modern neural network set- ting, however, it is not obvious how to use standard approaches like careful neural architecture design or data augmentation to induce such an inductive bias. In a way, while the shift from feature engi- neering towards end-to-end neural networks and representation learning has alleviated the burden of manual feature engineering and increased model expressivity, it has also reduced our control over the inductive biases of a model. In this paper, we explore using natural language explanations (Figure 1) to generate features that can augment modern neural representations. This imbues representations with inductive biases cor- responding to the explanations, thereby restoring some degree of control while maintaining their ex- pressive power. X Jim Bob Michelle Duggar y X Stephen Mel y X Captain Darren Fletcher Berahino y Explanations: Training Data: Figure 1: Sample data points and explanations from Spouse, one of our relation extraction tasks. The ex- planations provide relevant features for classification. Prior work on training models with explanations use semantic parsers to interpret explanations: the parser converts each explanation into an executable logical form that is executable over the input sen- tence and uses the resulting outputs as features (Srivastava et al., 2017) or as noisy labels on un- labeled data (Hancock et al., 2018). However, se- mantic parsers can typically only parse low-level statements like “‘wife’ appears between {o1} and {o2} and the last word of {o1} is the same as the last word of {o2}” (Hancock et al., 2018). We remove these limitations by using modern distributed language representations, instead of se- mantic parsers, to interpret language explanations. Our approach, ExpBERT (Figure 2), uses BERT (Devlin et al., 2019) fine-tuned on the MultiNLI natural language inference dataset (Williams et al., 2018) to produce features that “interpret” each ex- planation …

Content truncated to 500 words