Multilingual Language Processing From Bytes

Proceedings of NAACL-HLT 2016, pages 1296–1306, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Multilingual Language Processing From Bytes Dan Gillick, Cliff Brunk, Oriol Vinyals, Amarnag Subramanya Google Research {dgillick, cliffbrunk, vinyals, asubram}@google.com Abstract We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start posi- tions, lengths, and labels are separate entries in our vocabulary. Because we operate di- rectly on unicode bytes rather than language- specific words or characters, we can analyze text in many languages with a single model. Due to the small vocabulary size, these multi- lingual models are very compact, but produce results similar to or better than the state-of- the-art in Part-of-Speech tagging and Named Entity Recognition that use only the provided training datasets (no external data sources). Our models are learning “from scratch” in that they do not rely on any elements of the stan- dard pipeline in Natural Language Processing (including tokenization), and thus can run in standalone fashion on raw text. 1 Introduction The long-term trajectory of research in Natural Language Processing has seen the replacement of rules and specific linguistic knowledge with ma- chine learned components. Perhaps the most stan- dardized way that knowledge is still injected into largely statistical systems is through the processing pipeline: Some set of basic language-specific tokens are identified in a first step. Sequences of tokens are segmented into sentences in a second step. The resulting sentences are fed one at a time for syntac- tic analysis: Part-of-Speech (POS) tagging and pars- ing. Next, the predicted syntactic structure is typi- cally used as features in semantic analysis, Named Entity Recognition (NER), Semantic Role Labeling, etc. While each step of the pipeline now relies more on data and models than on hand-curated rules, the pipeline structure itself encodes one particular un- derstanding of how meaning attaches to raw strings. One motivation for our work is to try removing this structural dependence. Rather than rely on the intermediate representations invented for specific subtasks (for example, Penn Treebank tokenization), we are allowing the model to learn whatever internal structure is most conducive to producing the annota- tions of interest. To this end, we describe a Recur- rent Neural Network (RNN) model that reads raw in- put string segments, one byte at a time, and produces output span annotations corresponding to specific byte regions in the input1. This is truly language annotation from scratch (see Collobert et al. (2011) and Zhang and LeCun (2015)). Two key innovations facilitate this approach. First, Long Short Term Memory (LSTM) models (Hochreiter and Schmidhuber, 1997) allow us to re- place the traditional independence assumptions in text processing with structural constraints on mem- ory. While we have long known that long-term de- pendencies are important in language, we had no mechanism other than conditional independence to keep sparsity in check. The memory in an LSTM, however, is not constrained by any explicit assump- tions of …

Content truncated to 500 words