{"id":3184,"date":"2021-03-25T10:27:54","date_gmt":"2021-03-25T09:27:54","guid":{"rendered":"https:\/\/www.basyskom.de\/?p=3184"},"modified":"2021-03-25T10:35:40","modified_gmt":"2021-03-25T09:35:40","slug":"what-is-apache-avro-compared-to-protobuf","status":"publish","type":"post","link":"https:\/\/www.basyskom.de\/en\/what-is-apache-avro-compared-to-protobuf\/","title":{"rendered":"What is Apache Avro compared to Protobuf"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"3184\" class=\"elementor elementor-3184\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-1006e21 elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"1006e21\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-def7ee4\" data-id=\"def7ee4\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-c3eeea1 elementor-widget elementor-widget-text-editor\" data-id=\"c3eeea1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Introduction &#8211; Apache Avro and Protobuf<\/h2><p>After we have learned some details about Protobuf in the\u00a0<a href=\"https:\/\/www.basyskom.de\/2020\/protobuf-for-iot\" target=\"_blank\" rel=\"noopener\">second part<\/a>\u00a0of the IoT series, we will now give an introduction to\u00a0<a href=\"https:\/\/avro.apache.org\/\" target=\"_blank\" rel=\"noopener\">Apache Avro<\/a>\u00a0and work out how it compares to Protobuf and if it is suitable to be used in IoT devices.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-2c1c646 elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"2c1c646\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e223781\" data-id=\"e223781\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-b80aece elementor-widget elementor-widget-text-editor\" data-id=\"b80aece\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Apache Avro Basics<\/h2><p>Avro is a framework for data serialization which has been developed as part of <a href=\"https:\/\/hadoop.apache.org\/\" target=\"_blank\" rel=\"noopener\">Apache Hadoop<\/a>.<br \/>It is mainly used in big data processing and is also supported by some of the data processing solutions in Microsoft Azure.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-c5c4e5d elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"c5c4e5d\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-136dca7\" data-id=\"136dca7\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-084049b elementor-widget elementor-widget-text-editor\" data-id=\"084049b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Schemas<\/h3><p>Avro uses schemas to structure the data. Schemas are usually defined in <a href=\"https:\/\/avro.apache.org\/docs\/current\/spec.html#schema_primitive\" target=\"_blank\" rel=\"noopener\">JSON<\/a>, but there is also support for an <a href=\"http:\/\/avro.apache.org\/docs\/current\/idl.html\" target=\"_blank\" rel=\"noopener\">IDL<\/a>. This post will concentrate on the JSON format.<\/p><p>As an example, we will now recreate the environment sensor messages from the\u00a0<a href=\"https:\/\/www.basyskom.de\/2020\/protobuf-for-iot\" target=\"_blank\" rel=\"noopener\">Protobuf post<\/a>\u00a0as a JSON schema. Some changes are necessary due to differences between Protobuf and Avro<\/p><ul><li>Avro does not support <b>unsigned<\/b> types. The <code>timestamp<\/code> becomes a 64 bit <b>signed<\/b> integer.<\/li><li>Contrary to Protobuf, where all fields are <b>optional<\/b>, Avro does not support optional fields. Every field should get a <b>default<\/b> value to allow removing it when changing the schema.<\/li><li>Avro does not have Protobuf&#8217;s <b>oneof<\/b> feature. As a replacement, <code>TelemetryMessage<\/code> gets a field named payload whose type is an <b>union<\/b> of the two types <code>EnvironmentData<\/code> and <code>Event<\/code>.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1736afe elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"1736afe\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-json'>{\n  &quot;namespace&quot;: &quot;iotexample&quot;,\n  &quot;name&quot;: &quot;DeviceMessages&quot;,\n  &quot;type&quot;: &quot;record&quot;,\n  &quot;fields&quot;: [\n    {\n      &quot;name&quot;: &quot;telemetry_messages&quot;,\n      &quot;type&quot;: {\n        &quot;namespace&quot;: &quot;iotexample&quot;,\n        &quot;name&quot;: &quot;TelemetryMessagesArray&quot;,\n        &quot;type&quot;: &quot;array&quot;,\n        &quot;items&quot;: {\n          &quot;namespace&quot;: &quot;iotexample&quot;,\n          &quot;name&quot;: &quot;TelemetryMessage&quot;,\n          &quot;type&quot;: &quot;record&quot;,\n          &quot;fields&quot;: [\n            {\n              &quot;name&quot;: &quot;timestamp&quot;,\n              &quot;type&quot;: &quot;long&quot;,\n              &quot;default&quot;: 0\n            },\n            {\n              &quot;name&quot;: &quot;payload&quot;,\n              &quot;type&quot;: [\n                {\n                  &quot;namespace&quot;: &quot;iotexample&quot;,\n                  &quot;name&quot;: &quot;EnvironmentData&quot;,\n                  &quot;type&quot;: &quot;record&quot;,\n                  &quot;fields&quot;: [\n                    {\n                      &quot;name&quot;: &quot;temperature&quot;,\n                      &quot;type&quot;: &quot;double&quot;,\n                      &quot;default&quot;: 0\n                    },\n                    {\n                      &quot;name&quot;: &quot;pressure&quot;,\n                      &quot;type&quot;: &quot;double&quot;,\n                      &quot;default&quot;: 0\n                    },\n                    {\n                      &quot;name&quot;: &quot;humidity&quot;,\n                      &quot;type&quot;: &quot;double&quot;,\n                      &quot;default&quot;: 0\n                    },\n                    {\n                      &quot;name&quot;: &quot;co2_level&quot;,\n                      &quot;type&quot;: &quot;double&quot;,\n                      &quot;default&quot;: 0\n                    }\n                  ]\n                },\n                {\n                  &quot;namespace&quot;: &quot;iotexample&quot;,\n                  &quot;type&quot;: &quot;record&quot;,\n                  &quot;name&quot;: &quot;Event&quot;,\n                  &quot;fields&quot;: [\n                    {\n                      &quot;type&quot;: &quot;string&quot;,\n                      &quot;name&quot;: &quot;message&quot;,\n                      &quot;default&quot;: &quot;&quot;\n                    },\n                    {\n                      &quot;type&quot;: &quot;int&quot;,\n                      &quot;name&quot;: &quot;event_number&quot;,\n                      &quot;default&quot;: 0\n                    },\n                    {\n                      &quot;name&quot;: &quot;error_level&quot;,\n                      &quot;type&quot;: {\n                        &quot;namespace&quot;: &quot;iotexample&quot;,\n                        &quot;name&quot;: &quot;ErrorLevel&quot;,\n                        &quot;type&quot;: &quot;enum&quot;,\n                        &quot;symbols&quot;: [\n                          &quot;UNSPECIFIED&quot;,\n                          &quot;ERROR&quot;,\n                          &quot;WARNING&quot;,\n                          &quot;INFO&quot;\n                        ]\n                      },\n                      &quot;default&quot;: &quot;UNSPECIFIED&quot;\n                    }\n                  ]\n                }\n              ]\n            }\n          ]\n        }\n      }\n    }\n  ]\n}\n <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-81ac666 elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"81ac666\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-506bf6d\" data-id=\"506bf6d\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-89677eb elementor-widget elementor-widget-text-editor\" data-id=\"89677eb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Schema Evolution<\/h2><p>\nChanging\u00a0an existing Avro schema (removing fields, adding new fields, etc.) shows the main difference to Protobuf.\n<\/p><p>\nWe have learned that Protobuf messages can be evolved in a way that enables a consumer which only knows about the new version to consume messages created with the old version and vice versa.\n<\/p><p>\nIn Avro, this is not possible as the consumer must always know the schema that was used to serialize the message. There are different levels of compatibility which allow different changes and are explained <a href=\"https:\/\/docs.confluent.io\/current\/schema-registry\/avro.html\" target=\"_blank\" rel=\"noopener\">here<\/a>.\n<\/p><p>\nSystems using Avro usually employ a\u00a0<a style=\"background-color: #ffffff;\" href=\"https:\/\/docs.confluent.io\/current\/schema-registry\/index.html\" target=\"_blank\" rel=\"noopener\">schema registry<\/a>\u00a0where all versions of a schema are stored. Messages must then be prefixed with the identifier of the schema used by the producer to allow the consumer to decode the message.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-197c80c elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"197c80c\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-0f484ec\" data-id=\"0f484ec\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-5972b29 elementor-widget elementor-widget-text-editor\" data-id=\"5972b29\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Programming with Avro in C++<\/h2>\nTo demonstrate the C++ API, we\u00a0again create a Qt based application. This application uses the\u00a0<a href=\"https:\/\/github.com\/apache\/avro\/tree\/master\/lang\/c%2B%2B\" target=\"_blank\" rel=\"noopener\">Avro C++ library<\/a>\u00a0which must be built and installed.\n\nThe C++ version also provides a code generator named <code>avrogencpp<\/code>. We use it to create the message structs and the serialization and deserialization code.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-574a780 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"574a780\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-bash'>avrogencpp -i iotmessages.json -o iotmessages.h -n iotexample <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-969a7f3 elementor-widget elementor-widget-text-editor\" data-id=\"969a7f3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h4>Walkthrough<\/h4>\nFirst, we need some includes for Avro and the Qt features we are going to use\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0018d5f elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"0018d5f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-cpp'>#include &quot;iotmessages.h&quot;\n\n#include &lt;avro\/ValidSchema.hh&gt;\n#include &lt;avro\/Compiler.hh&gt;\n\n#include &lt;QCoreApplication&gt;\n#include &lt;QDateTime&gt;\n#include &lt;QDebug&gt;\n#include &lt;QFile&gt; <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-def4dac elementor-widget elementor-widget-text-editor\" data-id=\"def4dac\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Then we use the generated structs from <code>iotmessages.h<\/code> and populate a <code>DeviceMessages<\/code> message.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-812b49e elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"812b49e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-cpp'>int main()\n{\n    \/\/ Create and populate a TelemetryMessage with EnvironmentData payload\n    iotexample::TelemetryMessage sensorDataMessage;\n    sensorDataMessage.timestamp = QDateTime::currentMSecsSinceEpoch();\n\n    iotexample::EnvironmentData payload;\n    payload.temperature = 23;\n    payload.pressure = 1080;\n    payload.humidity = 75;\n    payload.co2_level = 415;\n\n    sensorDataMessage.payload.set_EnvironmentData(payload);\n\n    \/\/ Create and populate a TelemetryMessage with Event payload\n    iotexample::TelemetryMessage eventMessage;\n    eventMessage.timestamp = QDateTime::currentMSecsSinceEpoch();\n\n    iotexample::Event event;\n    event.message = std::string(&quot;My event message&quot;);\n    event.error_level = iotexample::ErrorLevel::ERROR;\n    event.event_number = 123;\n\n    eventMessage.payload.set_Event(event);\n\n    \/\/ Create a DeviceMessages message and add the two messages\n    iotexample::DeviceMessages messages;\n    messages.telemetry_messages.push_back(sensorDataMessage);\n    messages.telemetry_messages.push_back(eventMessage); <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3d9377c elementor-widget elementor-widget-text-editor\" data-id=\"3d9377c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>In the next step, we will serialize the message and print the serialized data size and the <code>base64<\/code> encoded serialized data.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2c3ddd9 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"2c3ddd9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-cpp'>    \/\/ Create output stream and encoder\n    auto out = avro::memoryOutputStream();\n    auto encoder = avro::binaryEncoder();\n    encoder-&gt;init(*out);\n\n    \/\/ Encode the message\n    avro::encode(*encoder, messages);\n    encoder-&gt;flush();\n\n    qDebug() &lt;&lt; &quot;Serialized size:&quot; &lt;&lt; out-&gt;byteCount() &lt;&lt; &quot;bytes&quot;;\n\n    \/\/ Create input stream and reader\n    auto in = avro::memoryInputStream(*out);\n    auto reader = avro::StreamReader(*in);\n\n    QByteArray data(out-&gt;byteCount(), Qt::Uninitialized);\n\n    reader.readBytes(reinterpret_cast&lt;quint8 *&gt;(data.data()), out-&gt;byteCount());\n\n    qDebug() &lt;&lt; &quot;Serialized data:&quot; &lt;&lt; data.toBase64(); <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-475feea elementor-widget elementor-widget-text-editor\" data-id=\"475feea\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>To demonstrate the decoder, we will now use a <code>validatingDecoder<\/code> to deserialize the message. This decoder is initialized with the schema and will throw an exception if the data doesn&#8217;t match.<\/p><p>\n\nAfter decoding the message, the content is printed. The printing code also shows how to handle the union we used as <code>oneof<\/code> replacement.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3d8792a elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"3d8792a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-cpp'>    \/\/ Load schema file from the resource\n    QFile schemaFile(&quot;:iotmessages.json&quot;);\n    schemaFile.open(QFile::ReadOnly);\n    auto schemaInput = std::istringstream(schemaFile.readAll().toStdString());\n\n    avro::ValidSchema schema;\n\n    try {\n        avro::compileJsonSchema(schemaInput, schema);\n    } catch(std::exception &amp;ex) {\n        qWarning() &lt;&lt; &quot;Failed to compile schema:&quot; &lt;&lt; ex.what();\n        return 1;\n    }\n\n    \/\/ Create input stream and a validating binary decoder\n    in = avro::memoryInputStream(*out);\n    auto decoder = avro::binaryDecoder();\n\n    decoder-&gt;init(*in);\n    auto validatingDecoder = avro::validatingDecoder(schema, decoder);\n\n    iotexample::DeviceMessages decoded;\n\n    try {\n        avro::decode(*validatingDecoder, decoded);\n    } catch(std::exception &amp;ex) {\n        qWarning() &lt;&lt; &quot;Decode failed with:&quot; &lt;&lt; ex.what();\n        return 1;\n    }\n\n    qDebug() &lt;&lt; &quot;Decoded message with&quot; &lt;&lt; decoded.telemetry_messages.size() &lt;&lt; &quot;values&quot;;\n\n    for (uint i = 0; i &lt; decoded.telemetry_messages.size(); ++i) {\n        const auto &amp;current = decoded.telemetry_messages.at(i);\n        qDebug() &lt;&lt; &quot;Message&quot; &lt;&lt; i + 1;\n        qDebug() &lt;&lt; &quot;  Timestamp:&quot; &lt;&lt; QDateTime::fromMSecsSinceEpoch(current.timestamp).toString(Qt::ISODate);\n        if (current.payload.idx() == 1) {\n            qDebug() &lt;&lt; &quot;  Event number:&quot; &lt;&lt; current.payload.get_Event().event_number;\n            qDebug() &lt;&lt; &quot;  Event error level:&quot; &lt;&lt; (int) current.payload.get_Event().error_level;\n            qDebug() &lt;&lt; &quot;  Event message:&quot; &lt;&lt; QString::fromStdString(current.payload.get_Event().message);\n        } else if (current.payload.idx() == 0) {\n            qDebug() &lt;&lt; &quot;  Temperature:&quot; &lt;&lt; current.payload.get_EnvironmentData().temperature;\n            qDebug() &lt;&lt; &quot;  Pressure:&quot; &lt;&lt; current.payload.get_EnvironmentData().pressure;\n            qDebug() &lt;&lt; &quot;  Humidity:&quot; &lt;&lt; current.payload.get_EnvironmentData().humidity;\n            qDebug() &lt;&lt; &quot;  CO2:&quot; &lt;&lt; current.payload.get_EnvironmentData().co2_level;\n        } else {\n            qDebug() &lt;&lt; &quot;  Empty TelemetryMessages&quot;;\n        }\n    }\n\n    return 0;\n} <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1d4d44f elementor-widget elementor-widget-text-editor\" data-id=\"1d4d44f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Building the example<\/h3><p>We use qmake to build the example application. The JSON schema is included as a resource and is also used to generate the C++ code at build time.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d543dc8 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"d543dc8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-solid'>QT -= gui\n\nCONFIG += c++11 console\nCONFIG -= app_bundle\n\nINCLUDEPATH += \/opt\/avro\/include\n\nLIBS += -L\/opt\/avro\/lib -lavrocpp\n\nHEADERS += \\\n    iotmessages.h\n\nSOURCES += \\\n        main.cpp\n\nAVRO_COMPILER = \/opt\/avro\/bin\/avrogencpp\ncommand = $$AVRO_COMPILER -i iotmessages.json -o iotmessages.h -n iotexample\n!system($$command) {\n    error(&quot;avrogencpp is required to build this application&quot;)\n}\n\nRESOURCES += \\\n    schema.qrc <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d0a4911 elementor-widget elementor-widget-text-editor\" data-id=\"d0a4911\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h4>Result<\/h4>\n<p>The application is executed and prints the result.\n<\/p><p>\nWe see that the serialized data is <b>68 bytes<\/b> in size, which is less than the <b>80 bytes<\/b> needed by Protobuf. The main cause is the lack of optional fields and field numbers in Avro messages.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8d72f64 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"8d72f64\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-solid'>Serialized size: 68 bytes\nSerialized data: &quot;BLCsqYnDXAAAAAAAAAA3QAAAAAAA4JBAAAAAAADAUkAAAAAAAPB5QLCsqYnDXAIgTXkgZXZlbnQgbWVzc2FnZfYBAgA=&quot;\nDecoded message with 2 values\nMessage 1\n  Timestamp: &quot;2020-05-15T15:45:28&quot;\n  Temperature: 23\n  Pressure: 1080\n  Humidity: 75\n  CO2: 415\nMessage 2\n  Timestamp: &quot;2020-05-15T15:45:28&quot;\n  Event number: 123\n  Event error level: 1\n  Event message: &quot;My event message&quot; <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-892e5e3 elementor-widget elementor-widget-text-editor\" data-id=\"892e5e3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Programming with Avro in\u00a0JavaScript \/ Node.js<\/h2>\nWe will now do the same encode\/decode routine in Node.js.\nThis example requires the\u00a0<a href=\"https:\/\/www.npmjs.com\/package\/avro-js\" target=\"_blank\" rel=\"noopener\">avro-js<\/a>\u00a0module.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-51fbee1 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"51fbee1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>const avro = require(&#039;avro-js&#039;);\n\n\/\/ Parse the schema and create the corresponding type\nconst deviceRecordMessage = avro.parse(&#039;.\/iotmessages.json&#039;);\n\n\/\/ Create a sample message\nconst messageToEncode = {\n    telemetry_messages: [\n        {\n            timestamp: Date.now(),\n            payload: {\n                &#039;iotexample.EnvironmentData&#039;: {\n                    temperature: 23,\n                    pressure: 1080,\n                    humidity: 75,\n                    co2_level: 415\n                }\n            }\n        },\n        {\n            timestamp: Date.now(),\n            payload: {\n                &#039;iotexample.Event&#039;: {\n                    message: &#039;My event message&#039;,\n                    event_number: 123,\n                    error_level: &#039;ERROR&#039;\n                }\n            }\n        }\n    ]\n}\n\n\/\/ Encode the message\nconst data = deviceRecordMessage.toBuffer(messageToEncode);\n\n\/\/ Print encoded data as base64\nconsole.log(`Encoded data: &quot;${data.toString(&#039;base64&#039;)}&quot;, length: ${data.length}`);\n\n\/\/ Decode the message\nconst decoded = deviceRecordMessage.fromBuffer(data);\n\n\/\/ Print the content of the decoded message\nconsole.log(JSON.stringify(decoded, null, 2)); <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-05e14e0 elementor-widget elementor-widget-text-editor\" data-id=\"05e14e0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Running the example of course shows the same encoded data as the C++ example. The decoded data is JSON and follows the form that can be expected from the schema.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d6e6a3a elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"d6e6a3a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-json'>Encoded data: &quot;BOKUk+vEXAAAAAAAAAA3QAAAAAAA4JBAAAAAAADAUkAAAAAAAPB5QOKUk+vEXAIgTXkgZXZlbnQgbWVzc2FnZfYBAgA=&quot;, length: 68\n\n{\n  &quot;telemetry_messages&quot;: [\n    {\n      &quot;timestamp&quot;: 1589787125041,\n      &quot;payload&quot;: {\n        &quot;iotexample.EnvironmentData&quot;: {\n          &quot;temperature&quot;: 23,\n          &quot;pressure&quot;: 1080,\n          &quot;humidity&quot;: 75,\n          &quot;co2_level&quot;: 415\n        }\n      }\n    },\n    {\n      &quot;timestamp&quot;: 1589787125041,\n      &quot;payload&quot;: {\n        &quot;iotexample.Event&quot;: {\n          &quot;message&quot;: &quot;My event message&quot;,\n          &quot;event_number&quot;: 123,\n          &quot;error_level&quot;: &quot;ERROR&quot;\n        }\n      }\n    }\n  ]\n} <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-061c55e elementor-widget elementor-widget-text-editor\" data-id=\"061c55e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Schema evolution in JavaScript<\/h3>\nWe will now have at look what happens if the consumer in the cloud uses an updated schema and receives a message encoded with an older schema.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3baf6cd elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"3baf6cd\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>const avro = require(&#039;avro-js&#039;);\nconst fs = require(&#039;fs&#039;)\n\nconst schema = JSON.parse(fs.readFileSync(&#039;.\/iotmessages.json&#039;))\n\n\/\/ Parse the schema and create the corresponding type\nconst deviceRecordMessage = avro.parse(schema);\n\n\/\/ Create a sample message\nconst messageToEncode = {\n    telemetry_messages: [\n        {\n            timestamp: Date.now(),\n            payload: {\n                &#039;iotexample.EnvironmentData&#039;: {\n                    temperature: 23,\n                    pressure: 1080,\n                    humidity: 75,\n                    co2_level: 415\n                }\n            }\n        },\n        {\n            timestamp: Date.now(),\n            payload: {\n                &#039;iotexample.Event&#039;: {\n                    message: &#039;My event message&#039;,\n                    event_number: 123,\n                    error_level: &#039;ERROR&#039;\n                }\n            }\n        }\n    ]\n}\n\n\/\/ Encode the message\nconst data = deviceRecordMessage.toBuffer(messageToEncode);\n\n\/\/ Print encoded data as base64\nconsole.log(`Encoded data: &quot;${data.toString(&#039;base64&#039;)}&quot;, length: ${data.length}`); <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9db21e5 elementor-widget elementor-widget-text-editor\" data-id=\"9db21e5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The first change will be to remove the <code>co2_level<\/code> field from the schema and then attempting to decode the encoded data with the new schema.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5c96265 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"5c96265\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>let decoded = null\n\n\/\/ Modify the schema, rename co2_level to ambient_light\nlet modifiedSchema = JSON.parse(JSON.stringify(schema));\nmodifiedSchema.fields[0].type.items.fields[1].type[0].fields[3].name = &#039;ambient_light&#039;;\nlet deviceRecordMessageEvolved = avro.parse(modifiedSchema);\n\ndecoded = deviceRecordMessageEvolved.fromBuffer(data);\nconsole.log(`${decoded.telemetry_messages[0].payload[&#039;iotexample.EnvironmentData&#039;].ambient_light} was the value of co2_level`) <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-92f211c elementor-widget elementor-widget-text-editor\" data-id=\"92f211c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Running the code shows that the value that was set in the <code>co2_level<\/code> field is now decoded as <code>ambient_light<\/code> because the semantic change is just in the schema and the encoded data has not changed..<\/p><p>The second modified schema adds the <code>ambient_light<\/code> field in addition to the <code>co2_level<\/code> field.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cdc2b60 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"cdc2b60\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>console.log(&quot;######## Try to decode a message with a newer type with a removed field ########&quot;);\n\ndecoded = deviceRecordMessageEvolved.fromBuffer(data);\nconsole.log(`${decoded.telemetry_messages[0].payload[&#039;iotexample.EnvironmentData&#039;].ambient_light} was the value of co2_level`)\n\n\/\/ Modify the schema, add field ambient_light in addition to co2_level\nmodifiedSchema = JSON.parse(JSON.stringify(schema));\nmodifiedSchema.fields[0].type.items.fields[1].type[0].fields.push(\n    {\n        name: &#039;ambient_light&#039;,\n        type: &#039;double&#039;,\n        defaul&quot;: 0\n    }\n);\ndeviceRecordMessageEvolved = avro.parse(modifiedSchema);\n\ntry {\n    decoded = deviceRecordMessageEvolved.fromBuffer(data);\n} catch(ex) {\n    console.error(`Failed to decode the message encoded with the old schema: ${ex}`);\n} <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5544841 elementor-widget elementor-widget-text-editor\" data-id=\"5544841\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The decoder now fails with <code>Error: invalid union index: -61<\/code> because the offsets expected by the decode assume an additional double value which is not present in the encoded data.<\/p><p>We now decode the message using a resolver which maps the encoded data to the new schema with the additional field.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6f56002 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"6f56002\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>\/\/ Create a resolving decoder and decode the message\nresolver = deviceRecordMessageEvolved.createResolver(deviceRecordMessage);\n\ndecoded = deviceRecordMessageEvolved.fromBuffer(data, resolver);\n\nconsole.log(JSON.stringify(decoded, null, 2)); <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-03c7734 elementor-widget elementor-widget-text-editor\" data-id=\"03c7734\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The data is now decoded as expected, the <code>ambient_light<\/code> field which is not present in the encoded data is set to the default value that was specified in the schema.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-046c5d6 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"046c5d6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-json'>{\n  &quot;telemetry_messages&quot;: [\n    {\n      &quot;timestamp&quot;: 1589784854837,\n      &quot;payload&quot;: {\n        &quot;iotexample.EnvironmentData&quot;: {\n          &quot;temperature&quot;: 23,\n          &quot;pressure&quot;: 1080,\n          &quot;humidity&quot;: 75,\n          &quot;co2_level&quot;: 415,\n          &quot;ambient_light&quot;: 0\n        }\n      }\n    },\n    {\n      &quot;timestamp&quot;: 1589784854837,\n      &quot;payload&quot;: {\n        &quot;iotexample.Event&quot;: {\n          &quot;message&quot;: &quot;My event message&quot;,\n          &quot;event_number&quot;: 123,\n          &quot;error_level&quot;: &quot;ERROR&quot;\n        }\n      }\n    }\n  ]\n} <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cf8c636 elementor-widget elementor-widget-text-editor\" data-id=\"cf8c636\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tTo show the importance of default values in the schema, we will now add the <code>ambient_light<\/code> field without specifying a default value and repeat the previous test\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ef5a81 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"0ef5a81\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-javascript'>modifiedSchema = JSON.parse(JSON.stringify(schema));\nmodifiedSchema.fields[0].type.items.fields[1].type[0].fields.push(\n    {\n        name: &#039;ambient_light&#039;,\n        type: &#039;double&#039;\n    }\n);\ndeviceRecordMessageEvolved = avro.parse(modifiedSchema);\n\ntry {\n    resolver = deviceRecordMessageEvolved.createResolver(deviceRecordMessage);\n} catch (ex) {\n    console.error(`Decode failed without default value: ${ex}`);\n} <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9cc2b27 elementor-widget elementor-widget-text-editor\" data-id=\"9cc2b27\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The attempt to create a resolver fails due to the missing default value<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9046574 elementor-widget elementor-widget-elementor-syntax-highlighter\" data-id=\"9046574\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"elementor-syntax-highlighter.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<pre><code class='language-solid'>Decode failed without default value: Error: cannot read &quot;iotexample.EnvironmentData&quot; as [&quot;iotexample.EnvironmentData&quot;,&quot;iotexample.Event&quot;] <\/code><\/pre><script>\nif (!document.getElementById('syntaxed-prism')) {\n\tvar my_awesome_script = document.createElement('script');\n\tmy_awesome_script.setAttribute('src','https:\/\/www.basyskom.de\/wp-content\/plugins\/syntax-highlighter-for-elementor\/assets\/prism2.js');\n\tmy_awesome_script.setAttribute('id','syntaxed-prism');\n\tdocument.body.appendChild(my_awesome_script);\n} else {\n\twindow.Prism && Prism.highlightAll();\n}\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-3ebf813 elementor-section-boxed elementor-section-height-default elementor-section-height-default wpr-particle-no wpr-jarallax-no wpr-parallax-no wpr-sticky-section-no wpr-column-slider-no wpr-equal-height-no\" data-id=\"3ebf813\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-a996ec2\" data-id=\"a996ec2\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-5a25619 elementor-widget elementor-widget-text-editor\" data-id=\"5a25619\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h2>Conclusion<\/h2><p>As we have seen from the example, an Avro message must always be prefixed with some information about which schema was used to encode it or the decoder will either fail or create invalid data. Adding default values to the schema is very important to allow a value to be removed later.<\/p><p>Despite the slightly smaller encoded data size for Avro, the ability to update Protobuf message definitions in a compatible way without having to prefix the encoded data with a schema identifier makes it a better choice for the data transmission of IoT devices.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Apache Avro is a framework for data serialization by Apache Hadoop. It is mainly used in big data processing but how does it compare to protobuf? Lets take a look.<\/p>","protected":false},"author":4,"featured_media":4651,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1,2,134,225],"tags":[237,137,136],"class_list":["post-3184","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-allgemein","category-blog","category-cloud","category-iot","tag-avro","tag-cloud","tag-iot"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/posts\/3184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/comments?post=3184"}],"version-history":[{"count":57,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/posts\/3184\/revisions"}],"predecessor-version":[{"id":4644,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/posts\/3184\/revisions\/4644"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/media\/4651"}],"wp:attachment":[{"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/media?parent=3184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/categories?post=3184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.basyskom.de\/en\/wp-json\/wp\/v2\/tags?post=3184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}