Weighted self Distillation for Chinese word segmentation

Findings of the Association for Computational Linguistics: ACL 2022, pages 139 - 149 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Table-based Fact Verification with Self-adaptive Mixture of Experts Yuxuan Zhou1, Xien Liu1, Kaiyin Zhou2,3, Ji Wu1 1Department of Electronic Engineering, Tsinghua University, Beijing 100084, China 2THiFLY Research, Tsinghua University, Beijing 100084, China 3State Key Laboratory of Cognitive Intelligence, Hefei, Anhui 230088, China Abstract The table-based fact verification task has re- cently gained widespread attention and yet re- mains to be a very challenging problem. It inherently requires informative reasoning over natural language together with different numer- ical and logical reasoning on tables (e.g., count, superlative, comparative). Considering that, we exploit mixture-of-experts and present in this paper a new method: Self-adaptive Mixture- of-Experts Network (SaMoE). Specifically, we have developed a mixture-of-experts neural net- work to recognize and execute different types of reasoning—the network is composed of mul- tiple experts, each handling a specific part of the semantics for reasoning, whereas a man- agement module is applied to decide the con- tribution of each expert network to the verifi- cation result. A self-adaptive method is devel- oped to teach the management module combin- ing results of different experts more efficiently without external knowledge. The experimental results illustrate that our framework achieves 85.1% accuracy on the benchmark dataset TAB- FACT, comparable with the previous state-of- the-art models. We hope our framework can serve as a new baseline for table-based ver- ification. Our code is available at https: //github.com/THUMLP/SaMoE. 1 Introduction Fact Verification, aiming to determine the consis- tency between a statement and given evidence, has become a crucial part of various applications such as fake news detection, rumor detection (Rashkin et al., 2017; Thorne et al., 2018; Goodrich et al., 2019; Vaibhav et al., 2019; Kryscinski et al., 2020). While most existing research focuses on verifica- tion based on unstructured text, a new trend is ex- tending the scope to structured evidence (e.g., ta- bles), which is informative and ubiquitous in our daily lives. Table-based verification faces different challenges than unstructured-text-based due to the complexity of the requirements, including sophis- ticated textual, numerical, and logical reasoning across evidence tables; even for some statements, multiple types of reasoning are indispensable to complete the verification. An example is presented in Figure 1. Figure 1: An Example of table-based fact verification. To tackle the challenges above, previous work established two kinds of methods: (1) program- enhanced methods (Chen et al., 2020; Zhong et al., 2020; Shi et al., 2020; Yang et al., 2020) and (2) table-based pre-trained models (Eisenschlos et al., 2020; Liu et al., 2021). The program-enhanced methods mainly leverage programs generated by the semantic parser. Specifically, statements are parsed into executable programs to extract the logi- cal/numerical semantics, which is further be lever- aged together with contextual semantics learned by a language model (e.g., BERT) in inference. How- ever, the semantic parsers that generate semantic- consistent programs must be trained in a weak su- pervision setting, which brings difficulties in train- ing. Furthermore, …

Content truncated to 500 words