Protocol Buffer Concepts

Internal

• Protocol Buffers

Overview

The main use case for Protocol Buffers is sharing data across programming languages. Data can be written and serialized in one language, sent over the network and then and deserialized and interpreted in a different programming language, without compatibility problems.

Protocol Buffers comes with the following advantages:

• It allows defining types, and the data is fully typed when exchanged. We know the type of data in transit.
• Data is compressed automatically.
• Serialization/deserialization is efficient.
• Comes with a schema (the .proto) file, which is used to generate code that writes and reads the data.
• Schema supports embedded documentation.
• Schema can evolve over time in a safe manner. The implementation can be maintained to be backward and forward compatible.

One of the disadvantages is that the data is encoded in a binary format, so it can't be visualized with a text editor.

The typical workflow consist in defining the data types, called messages, then automatically generating the data structures to support and validate the data types, in the programming language of choice. In Go, the messages are represented as structs. With the help of a framework like gRPC, which uses Protocol Buffers as default serialization format and native mechanism to exchange data, client and server code can also be automatically generated. This renders Protocol Buffer convenient for use as serialization format for microservices.

The current version of the protocol is 3, released mid 2016.

Message

A message is a data type in Protocol Buffers. One or more messages are declared in .proto text file with the following format:

syntax = "proto3";

message Person {
  int32 age = 1;
  string first_name = 2;
  string last_name = 3;
  bytes small_picture = 4;
  bool is_profile_verified = 5;
  float height = 6;
}

A message has fields.

Fields

Each field has a field type, a field name and a field tag

 <field_type> <field_name> = <field_tag>;

Field Names

Fields names are not important when the message is serialized/deserialized. Only the tags matter.

Field Types

Protocol Buffer Types

Field Tags

A tag is an integral value between 1 and 2²⁹-1 (536,879,911). The numbers between 19,000 and 19,999 are reserved and cannot be used.

Tags from 1 to 15 use one byte of space, so use them for frequently populated fields. The tags from 16 to 2047 use 2 bytes.

Data Type Support Code Generation

The code that supports the data types defined in the .proto files are generated as follows:

Data Evolution with Protocol Buffers

A message is actually a type. "Message" is used probably because the instances of the types defines as such are mainly intended to be sent over the wire.