Large Sequence Representation Learning via Multi-Stage Latent Transformers

Proceedings of the 29th International Conference on Computational Linguistics, pages 4633–4639 October 12–17, 2022. 4633 Large Sequence Representation Learning via Multi-Stage Latent Transformers Ionut-Catalin Sandu Daniel Voinea Alin-Ionut Popa Amazon Inc. {sanion, dvoinea, popaaln}@amazon.com Abstract We present LANTERN, a multi-stage trans- former architecture for named-entity recogni- tion (NER) designed to operate on indefinitely large text sequences (i.e. ≫ 512 elements). For a given image of a form with structured text, our method uses language and spatial fea- tures to predict the entity tags of each text el- ement. It breaks the quadratic computational constraints of the attention mechanism by op- erating over a learned latent space representa- tion which encodes the input sequence via the cross-attention mechanism while having the multi-stage encoding component as a refine- ment over the NER predictions. As a proxy task, we propose RADAR, an LSTM classifier operating at character level, which predicts the relevance of a word with respect to the entity- recognition task. Additionally, we formulate a challenging novel NER use case, nutritional information extraction from food product la- bels. We created a dataset with 11, 926 images depicting food product labels entitled TREAT dataset, with fully detailed annotations. Our method achieves superior performance against two competitive models designed for long se- quences on the proposed TREAT dataset. I. Introduction. Information extraction from im- ages and unstructured text plays a key role in natu- ral language understanding (NLU) with direct im- pact in domains such as news content synthesis (Oz- soy et al., 2011; Moratanch and Chitrakala, 2017; Foong et al., 2015), query search (Choi et al., 2018) or knowledge-base systems (Agirre et al., 2018). The transformer architecture (Vaswani et al., 2017) played a pivotal role in advancing the state-of-the art due to its robustness with respect to tasks re- lated to sequential data manipulation and under- standing. Its success relies mostly in the attention mechanism, considered as a key component for its versatility and its ability to self-specialize and filter the input information flow. At the same time, the attention is a curse for transformers, as it scales quadratically with respect to the size of the input sequence. To overcome this limitation, we propose a sequence parsing frame- work for textual information understanding allow- ing for indefinitely large input sequences, with focus on the NER task. Our method is entitled LANTERN (i.e. LArge sequeNce TransformER for NER). This is achieved via a multi-stage trans- former approach over a latent space representation of the input sequence inspired from (Jaegle et al., 2021a) and adapted to NER. The key innovation we propose is a framework which analyses the entire sequence via a latent space attention-based mod- elling and leverages a multi-stage prediction setup. II. Related Work. The transformer based archi- tecture led to significant advances across major areas of NLU such as text classification (Lin et al., 2021), question answering (Choi et al., 2018), se- quence to sequence translation (Rae et al., 2019; Lewis et al., 2019) and sentiment analysis (Zhang et al., 2018). Pretraining …

Content truncated to 500 words