Understanding Transformers for Information Extraction with Limited Data

Understanding Transformers for Information Extraction with Limited Data Minh-Tien Nguyen CINNAMON LAB, 10th floor, Geleximco building, 36 Hoang Cau, Dong Da, Hanoi, Vietnam. Hung Yen University of Technology and Education Hung Yen, Vietnam. tiennm@utehy.edu.vn Dung Tien Le, Nguyen Hong Son, Bui Cong Minh, Do Hoang Thai Duong, Le Thai Linh CINNAMON LAB, 10th floor, Geleximco building, 36 Hoang Cau, Dong Da, Hanoi, Vietnam. {nathan,levi,matt,howard}@cinnamon.is linh.le@uq.edu.au Abstract Transformers have recently achieved promis- ing results in many natural language process- ing tasks; however, the understanding of trans- formers for information extraction in business scenarios is still an open question. This paper bridges the gap by introducing an investiga- tion to understand the behavior of transform- ers in extracting information from domain- specific business documents. To do that, we employ transformers for taking advantage of these architectures trained on a huge amount of general data and fine-tune transformers to our down-stream IE task by using trans- fer learning. Experimental results on three Japanese datasets show that there are small margins among transformers in terms of F- scores but some models can achieve high ac- curacy with a small number of training data. 1 Introduction The significant growth of data provides a chance for humans to approach information from many sources. Yet, it also makes an obstacle for distilling useful knowledge. To address this issue, information extraction (IE) can be considered as an appropri- ate solution for converting unstructured to structured data. From the research side, due to its large impact, IE has received attention from the research commu- nity with many studies (Corro and Gemulla, 2013; Angeli et al., 2015; Nguyen et al., 2019). From the business site, IE is a crucial step for digital transfor- mation (Inmon and Nesavich, 2007; Herbert, 2017; Lin et al., 2019). The outputs of IE systems can be used in many natural language processing (NLP) ap- plications, e.g. question answering, information re- trieval (Shimaoka et al., 2016), or the automatic gen- eration of ontology (Fleischman and Hovy, 2002). The recent success of transformers draws a new direction for many NLP tasks. For example, BERT (Devlin et al., 2019) pioneers to creating a contex- tual language model for language understanding. As a result, BERT has achieved promising results on many NLP tasks, including IE. Following the suc- cess of BERT, a lot of transformer architecture has developed such as ALBERT (Lan et al., 2020), Dis- tillBERT (Sanh et al., 2019), or ELECTRA (Clark et al., 2019). It leverages the adaptation of trans- formers for IE. For example, (Nguyen et al., 2019) adapted BERT to extract information from business documents. These studies achieved promising re- sults; however, we argue that there exist gaps that limit the understanding of transformers for IE from domain-specific business documents. The first gap is that previous work only investigates the IE task with one transformer model, e.g. BERT. The second gap is that several important aspects of transformers were not studied well, e.g. the relationship between the number of training samples and performance. This …

Content truncated to 500 words