Universal Language Model Fine-tuning for Patent Classification

Universal Language Model Fine-tuning for Patent Classification Jason Hepburn Macquarie University Sydney, Australia jason.hepburn@students.mq.edu.au Abstract This paper describes the methods used for the 2018 ALTA Shared Task. The task this year was to automatically classify Australian patents into their main Interna- tional Patent Classification section. Our final submission used a Support Vector Machine (SVM) and Universal Language Model with Fine-tuning (ULMFiT). Our system achieved the best results in the stu- dent category. 1 Introduction For the last nine years the Australasian Language Technology Association (ALTA) has run a shared task competition for students. This year the shared task is to classify patent applications into their pri- mary section code (Moll´a and Seneviratne, 2018). Patent applications are classified and compared to previous inventions in the field. Accurate classi- fication of patents is crucial to patent officers, po- tential inventors, and industry. The patent classifi- cation process is dependant on human labour and with the rate of submissions increasing there is an ever greater need for an Automated Patent Classi- fication (APC) system (Fall et al., 2003). The International Patent Classification (IPC) has a tree structured class hierarchy (Silla and Fre- itas, 2011). At the highest level of this hierarchy is the IPC Section designated by the capital let- ters A to H (Table 1). Following the tree struc- ture from Sections are Classes, Sub-classes, and Groups. There are approximately 69,000 differ- ent categories at the group level. The classification taxonomy is revised annually and previous patents can be reclassified (D’hondt et al., 2013). Most patents have a main code in addition to a set of secondary codes. These secondary codes can be very distant to each other. For some codes A Human necessities B Performing operations, transporting C Chemistry, metallurgy D Textiles, paper E Fixed constructions F Mechanical engineering, lighting, heating, weapons, blasting G Physics H Electricity Table 1: IPC Sections it is obligatory to also assign other codes (eg. All C12N are also classed A61P). Codes can have placement rules defining a preference for one code when two may apply. At the semantic level all patents are different as they must describe a new idea or invention. Some terms, phrases, or acronyms can have very differ- ent meaning in different fields. Applicants try to avoid narrowing the scope of the invention and as such can use vague or general terms. As an ex- ample, pharmaceutical companies tend to describe every possible therapeutic use for an application. This can make it difficult to classify these patents. We structure this paper as follows: Section 2 introduces related research for APC; Section 3 de- scribes the data set provided for the competition; Section 4 describes the methods used; Section 5 presents and discusses the results; Section 6 con- cludes this paper. 2 Related works With the need for reliable and efficient APC sys- tems considerable research has been conducted in this area. Fall et al. (2003) introduce the publicly avail- able WIPO-alpha data set for patent classification (See section 3.2). They give a …

Content truncated to 500 words