Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 890–899 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 890 Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes No´emien Kocher♦ Christian Scuito♠ Lorenzo Tarantino♠ Alexandros Lazaridis♥ Andreas Fischer♦,♣ Claudiu Musat♥ ♦School of Engineering and Architecture of Fribourg, Switzerland, HES-SO, iCoSys Institute ♠Ecole Polytechnique F´ed´erale de Lausanne (EPFL) ♥Swisscom ♣University of Fribourg {noemien.kocher|sciutochristian|lorenzotara7}@gmail.com {alexandros.lazaridis|claudiu.musat}@swisscom.ch, andreas.fischer@hefr.ch Abstract In sequence modeling tasks the token order matters, but this information can be partially lost due to the discretization of the sequence into data points. In this paper, we study the imbalance between the way certain to- ken pairs are included in data points and oth- ers are not. We denote this a token order imbalance (TOI) and we link the partial se- quence information loss to a diminished per- formance of the system as a whole, both in text and speech processing tasks. We then pro- vide a mechanism to leverage the full token order information—Alleviated TOI—by iter- atively overlapping the token composition of data points. For recurrent networks, we use prime numbers for the batch size to avoid re- dundancies when building batches from over- lapped data points. The proposed method achieved state of the art performance in both text and speech related tasks. 1 Introduction Modeling sequences is a necessity. From time se- ries (Connor et al., 1994; Lane and Brodley, 1999) to text (Sutskever et al., 2011) and voice (Robin- son, 1994; Vinyals et al., 2012), ordered sequences account for a large part of the data we process and learn from. The data are discretized and become, in this paradigm, a list of tokens. The key to processing these token sequences is to model the interactions between them. Tradition- ally (Rosenfeld, 2000) this has been achieved with statistical methods, like N-grams. With the advances in computing power and the rebirth of neural networks, the dominant paradigm has become the use of recurrent neural networks (RNNs) (Mikolov et al., 2010). The dominance of RNNs has been recently challenged with great success by self-attention based models (Vaswani et al., 2017). Instead Contiguous tokens Data-point Order knowledge lost Figure 1: The common way of building data points given a dataset of contiguous tokens. Here we illus- trate a dataset with a contiguous list of 13 tokens, from which we build 3 data points of 4 tokens each. This process keeps the order of the tokens inside the data points, but loses the order information from token pairs that happen to fall between adjacent data points. of modeling the sequence linearly, Transformer- based models use learned correlations within the input to weight each element of the input sequence based on their relevance for the given task. Series discretization. Both RNNs and self- attention models take as input data points—token sequences of a maximum predefined length—and then create outputs for each of them. These tend to be much shorter in size, compared to the size of the full dataset. …

Content truncated to 500 words