Building a Custom Binary Format to Shrink JSON Payloads by 80%

Motivation for Binary Packing

The project began as an exploration into efficient data storage formats, inspired by discussions questioning the common practice of using JSON. The goal was to develop a method for data serialization that could offer significant size reductions compared to text-based formats like JSON.

Developing a Custom Binary Format

The developer created a binary schema language to address the inefficiencies of JSON. This involved designing a system to represent different data types and their lengths within a binary stream, moving beyond simple ASCII representations of numbers and strings.

Binary Encoding Principles

The core concept involves encoding data directly into binary, where each character or number occupies only the necessary bytes. For example, the ASCII value for 'h' (104) takes one byte, unlike its string representation '104' which would take three bytes. The custom format introduces type and length headers to allow a decoder to correctly interpret the packed binary data.

Achieving Payload Reduction

By implementing this custom binary packing method, the developer achieved an 80% reduction in JSON payload sizes. This efficiency gain comes from avoiding the verbose nature of text-based formats and directly encoding data types and values in their most compact binary form.