Why We Built Our Own Data Format

● By 2020, the gigabyte (GB) equivalent of all movies ever made will cross the global Internet every 2 minutes.

● Globally, IP traffic will reach 511 terabits per second (Tbps) in 2020, the equivalent of 142 million people streaming Internet high-definition (HD) video simultaneously, all day, every day.

● Global IP traffic by 2020 will be equivalent to 504 billion DVDs per year, 42 billion DVDs per month, or 58 million DVDs per hour.

Building smarter, scalable and fault tolerant applications that handle such a high volume of data is a huge challenge, but also represents a big opportunity for both startups and consumers. For disruptive startups (e.g. Blochchain startups) addressing big markets such as financial services, it puts them in a strong position to challenge the big incumbent financial institutions that mostly rely on centralised legacy technologies that were not designed for the current 24/7 explosion of connectivity and big data generated by consumers. For consumers, it opens up more choice and better quality of services at more competitive pricing than with the big incumbents.

Why do we need a new data format?

A couple years ago the benefits of distributed computing seemed to be only centered around the use case of academic research. A common use case often cited is the ability of scientists tackling hard scientific problems in fields such as genomics via cross disciplinary/institutional collaboration using grid computing to easily perform huge data processing/analysis tasks that otherwise would take months to accomplish.

1) A fast data format is of course faster to read and write (deserialize and serialize) for the communicating nodes. We also wanted something that can be traversed in its binary form if developers need maximum speed.

2) A compact data format requires less bytes to represent the encoded data. Fewer bytes requires less network bandwidth and can thus be transferred faster across the network.

Finally, let’s take the example of JSON which is currently a very popular data format that is widely used by developers. In fact so popular that it is common for companies to make their APIs JSON only. But JSON also has shortcomings including the following;

i) JSON is not a good format for raw binary data. Raw bytes must be Base64 or Hex encoded and transferred as strings. Base64 encoding increases the size of the encoded data to 4/3 of the raw size, and Hex encoding increases the size to 2/1 of the raw size.

ii) JSON is not that versatile in the sense that it is not that good at modelling all types of data structures. For example JSON is weak at modelling tables of similar data with rows and columns (e.g. CSV files). JSON would encode such tabular data as arrays of objects, meaning the column name would be repeated for every single object (row) in the table. This is a clear waste of data.

iii) JSON is not the fastest data format to read or write. Being verbose it is also slower to transfer, especially for devices with limited bandwidth like small IoT devices, mobile phones on weak connections or ships floating in the middle of the ocean.

What Is ION?

Before proceeding let us clarify something that otherwise may cause confusion. We published our ION data format well before Amazon Web Services publicly published their format also called ION! Therefore we did not choose the name ION because of its similarity to Amazon’s ION. In fact we did not discover the similarities until after we had designed and named our ION format. The acronym ION derives from “IAP Object Notation” where our open network protocol IAP stands for “Internet Application Protocol”. A straightforward evidence is the following;

In short, our ION is a versatile binary data format that can be used to encode a wide variety of data. It is expressive enough to contain serialized objects (e.g. Java or C# objects), CSV, JSON, XML, text and binary data. It is very fast and reasonably easy to parse and generate, more compressed on the wire than JSON and XML, and easy to handle for servers and routers and other lightweight hardware (we believe).

Being a data format ION can be used independently of our network protocol IAP. Developers can use ION as a data format in data files, log files, as data format for binary messages transmitted over HTTP etc. It can contain binary data so developers can also embed other formats inside when necessary e.g. an MP3 file, ZIP file, JPG file etc.

Nanosai Data Streaming Survey

Sensor City UK

Related posts:

LocalBitcoins is a bitcoin startup company based in Helsinki, Finland. Its service facilitates over-the-counter trading of local currency for bitcoins. Users post advertisements on the website, where…

I bought the coffee hole, that is its name, from my uncle several months ago. I worked for him my first semester of college after high school and then, well, you know, the pandemic hit. It’s gotten…

There are a variety of weight loss strategies and programs that are efficient, but the most effective option is dependent on your personal requirements and situation. A few general guidelines for…