Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4094–4103, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4094 Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control Mo Yu⇤ ~ Shiyu Chang⇤ |~ Yang Zhang⇤ |~ Tommi S. Jaakkola ♠ ~ IBM Research | MIT-IBM Watson AI Lab ♠ MIT yum@us.ibm.com {shiyu.chang, yang.zhang2}@ibm.com tommi@csail.mit.edu Abstract Selective rationalization has become a com- mon mechanism to ensure that predictive mod- els reveal how they use any available features. The selection may be soft or hard, and iden- tifies a subset of input features relevant for prediction. The setup can be viewed as a co- operate game between the selector (aka ratio- nale generator) and the predictor making use of only the selected features. The co-operative setting may, however, be compromised for two reasons. First, the generator typically has no direct access to the outcome it aims to justify, resulting in poor performance. Second, there’s typically no control exerted on the information left outside the selection. We revise the overall co-operative framework to address these chal- lenges. We introduce an introspective model which explicitly predicts and incorporates the outcome into the selection process. Moreover, we explicitly control the rationale complement via an adversary so as not to leave any useful information out of the selection. We show that the two complementary mechanisms maintain both high predictive accuracy and lead to com- prehensive rationales.1 1 Introduction Rapidly expanding applications of complex neu- ral models also bring forth criteria other than mere performance. For example, medical (Yala et al., 2019) and other high-value decision appli- cations require some means of verifying reasons for the predicted outcomes. This area of self- explaining models in the context of NLP appli- cations has primarily evolved along two parallel tracks. On one hand, we can design neural ar- chitectures that expose more intricate mechanisms ⇤Authors contributed equally to this work. 1The code and data for our method is publicly available at https://github.com/Gorov/three_player_ for_emnlp. Label: negative Original Text: really cloudy , lots of sediment , washed out yellow color . looks pretty gross , actually , like swamp water . no head , no lacing . Rationale from (Lei et al., 2016): [“really cloudy lots”, “yellow”, “no”, “no”] Rationale from cooperative introspection model: [“. looks”, “no”, “no”] Rationale from our full model: [“cloudy”, “lots”, “pretty gross”, “no lacing”] Table 1: An example of the rationales extracted by different models on the sentiment analysis task of beer reviews (ap- pearance aspect). Red words are human-labeled rationales. Details of the experiments can be found in Appendix B. of reasoning such as module networks (Andreas et al., 2016a,b; Johnson et al., 2017). While impor- tant, such approaches may still require adopting specialized designs and architectural choices that do not yet reach accuracies comparable to black- box approaches. On the other hand, we can im- pose limited architectural constraints in the form of selective rationalization (Lei et …

Content truncated to 500 words