UTF-8000 Specification Extends UTF-8 for Arbitrarily Large Code Units

Extending UTF-8 Capabilities

The UTF-8000 specification introduces a method to extend the UTF-8 encoding scheme, enabling the creation of arbitrarily large code units. This is achieved by providing clarity on how to split the highest bits of the first byte of UTF-8 code units into self-synchronization bits and start bits. The specification then details how to stripe these start bits across continuation bytes as needed.

Anatomy of UTF-8000 Code Units

An example 22-byte code unit demonstrates the power of UTF-8000, featuring multiple start bytes. The specification outlines the anatomical structure, including how continuation bytes can also function as start bytes and contain content bits, some of which may be straddled across different bytes.

Inherited Special Cases

UTF-8000 maintains only two special cases, both inherited directly from the original UTF-8 specification. These include the 1-byte ASCII encoding, which has 7 content bits and no mandatory content bits, and the 2-byte UTF-8 encoding, which has 4 mandatory content bits. A key aspect of UTF-8000 is that it introduces no new special cases in its extension of UTF-8.

Predictable Bit Counts

The number of content bits and mandatory content bits in UTF-8000 are predictable functions of the code unit's length. This predictability is a core property of the new specification, ensuring consistency in how character data is encoded and decoded.