1-0 Transformation Form of UTF-8

1-0 Transformation Form of UTF-8 Shengyuan Wu Shandong University, Shandong 250061, China Email: wsy@sdu.edu.cn Abstract. Based on Multilevel Mark Theory，11-10 and 1-0 transformation form of UTF-8 are proposed in this paper. The transformation between UCS and 1-0 form of UTF-8 is introduced, then, the transformation between Local Code and 1-0 Form of UTF-8 is discussed in detail. Keywords: Multilevel Mark theory; Unicode; UTF-8; UNIX 1. Introduction Based on Multilevel Mark Theory(1, 2)，a mixture object search method is realized, it can search various objects just like searching characters, and raise Chinese text search to word level. (3, 4) UTF-8 provides a UNIX compatible transformation format of Unicode Code System (UCS), and makes UNIX systems support multi-lingual text in a single encoding with multi-bytes. The mechanism of realizing UTF-8 influences the average length of UTF-8 and the transformation efficiency. The original form of UTF-8 （5, 6）was proposed in 1992, since then not any other forms have been proposed. In the original proposal of UTF-8, the first byte of multi-byte code determines the number of bytes of the code by setting the number of bits in high order to 1 and following a 0 bit; and the high-order bit of other byte is set to 1 and following a 0 bit. So, this form can be called as 110-10 form of UTF-8. Based on Multilevel Mark Theory(1, 2)，110-10 form of UTF-8 can be simplified to 11-10 form of UTF-8, and 11-10 form of UTF-8 can be further simplified to 1-0 form of UTF-8. 2. 11-10 transformation form of UTF-8 The proposal of 11-10 form of UTF-8 is shown in Table 1. It is easy to differentiate the bytes of a code, and to differentiate codes in a code sequence. Although the byte number of a code is not marked in the first byte, it is easy to count it. Table 1. Illustrating the 11-10 form of UTF-8 UCS Free bits 11-10 UTF-8 sequence U-00000000 - U-0000007F: 7 0xxxxxxx U-00000080 - U-00000FFF: 12 11xxxxxx 10xxxxxx U-00001000 - U-0003FFFF: 18 11xxxxxx 11xxxxxx 10xxxxxx U-00040000 - U-00FFFFFF: 24 11xxxxxx 11xxxxxx 11xxxxxx 10xxxxxx U-01000000 - U-3FFFFFFF: 30 11xxxxxx 11xxxxxx 11xxxxxx 11xxxxxx 10xxxxxx U-40000000 - U-7FFFFFFF: 36 11xxxxxx 11xxxxxx 11xxxxxx 11xxxxxx 11xxxxxx 10xxxxxx Take following code sequence as an example. 0xxxxxxx11xxxxxx 10xxxxxx 0xxxxxxx 11xxxxxx 11xxxxxx 10xxxxxx 354 Scanning the sequence from left to right, the first byte with mark bit 0 in the first bit is an ASCII code, the second byte is with mark “11”, so this is the first byte of a multi-byte code, the third byte is with mark “10”, so this is the end of the multi-byte code. The fourth byte is an ASCII code. The last 3 bytes is a 3 byte code. A character code in a 11-10 form of UTF-8 code sequence starts with a byte marked with “11”, ended with a byte marked with “10”, or starts with a byte marked with “0” and ended this byte. It is easy to make the transformation programs by the data structure as …

Content truncated to 500 words