Instance-Based Neural Dependency Parsing

Instance-Based Neural Dependency Parsing Hiroki Ouchi1,3 Jun Suzuki2,3 Sosuke Kobayashi2,4 Sho Yokoi2,3 Tatsuki Kuribayashi2,5 Masashi Yoshikawa2,3 Kentaro Inui2,3 1 NAIST, Japan 2 Tohoku University, Japan 3 RIKEN, Japan 4 Preferred Networks, Inc., Japan hiroki.ouchi@is.naist.jp,5 Langsmith, Inc., Japan sosk@preferred.jp, {jun.suzuki, yokoi, kuribayashi, yoshikawa, inui}@tohoku.ac.jp Abstract Interpretable rationales for model predictions are crucial in practical applications. We de- velop neural models that possess an interpret- able inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a train- ing set. The training edges are explicitly used for the predictions; thus, it is easy to grasp the contribution of each edge to the predictions. Our experiments show that our instance-based models achieve competitive accuracy with stan- dard neural models and have the reasonable plausibility of instance-based explanations. 1 Introduction While deep neural networks have improved pre- diction accuracy in various tasks, rationales under- lying the predictions have been more challenging for humans to understand (Lei et al., 2016). In practical applications, interpretable rationales play a crucial role in driving humans’ decisions and promoting human–machine cooperation (Ribeiro et al., 2016). From this perspective, the utility of instance-based learning (Aha et al., 1991), a traditional machine learning method, has been realized again (Papernot and McDaniel, 2018). Instance-based learning is a method that learns similarities between training instances and infers a value or class for a test instance on the basis of similarities against the training instances. On the one hand, standard neural models encode all the knowledge in the parameters, making it challeng- ing to determine what knowledge is stored and used for predictions (Guu et al., 2020). On the other hand, models with instance-based inference explicitly use training instances for predictions and can exhibit the instances that significantly contribute to the predictions. The instances play a role in an explanation to the question: why did the model make such a prediction? This type of explanation is called instance-based explana- tion (Caruana et al., 1999; Baehrens et al., 2010; Plumb et al., 2018), which facilitates the users’ understandings of model predictions and allows users to make decisions with higher confidence (Kolodneer, 1991; Ribeiro et al., 2016). It is not trivial to combine neural networks with instance-based inference processes while keep- ing high prediction accuracy. Recent studies in image recognition seek to develop such meth- ods (Wang et al., 2014; Hoffer and Ailon, 2015; Liu et al., 2017; Wang et al., 2018; Deng et al., 2019). This paradigm is called deep metric learn- ing. Compared to image recognition, there are much fewer studies on deep metric learning in natural language processing (NLP). As a few ex- ceptions, Wiseman and Stratos (2019) and Ouchi et al. (2020) developed neural models that have an instance-based inference process for sequence labeling tasks. They reported that their models have high explainability without sacrificing the prediction accuracy. As a next step from targeting consecutive to- kens, we study instance-based neural models for relations between discontinuous elements. To cor- …

Content truncated to 500 words