Protocol Buffers Data Type Go Code Generation

External

https://protobuf.dev/reference/go/go-generated

Internal

• Protocol Buffer Concepts

Overview

Protocol Buffer files are processed by the Protocol Buffer compiler protoc, which needs the Go plugin protoc-gen-go to generate Go code. The plugin is installed as described in the Installation section below. The protoc command line details are discussed in the Code Generation section.

Installation

The generic compiler must be installed with:

brew install protobuf

To generate Go code, the Go plugin must be installed with:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

This will install a protoc-gen-go binary in ${GOBIN}.

Code Generation

The compiler requires a mandatory source directory, specified with --proto_path, where it expects the find the .proto Protocol Buffer files, possibly scattered over a directory sub-hierarchy. The Protocol Buffer files are individually specified on the compiler's command line, and their paths must be relative to --proto_path.

The compiler creates one single Go source file for each .proto input file. The name of the output file is the name of the corresponding .proto file where the .proto extension is replaced with .pb.go. For example, a person.proto Protocol Buffer file will generate a person.pb.go Go source file.

The compiler must know Go package's import path for every .proto it processes, including those transitively depended upon by the .proto files being processed. Is the "import path" mentioned here the import path for the generated Go package, or something else? Is there an "import path" concept for the Protocol Buffer files themselves? There are two ways to specify the Go package import path:

• By declaring it within the .proto file with option go_package. This method is recommended and described in detail in the Declare Package Input Path inside the .proto Files section.
• By declaring it on the command line when invoking protoc using --go_opt options.

The newly created Go source files will be placed somewhere in the root directory specified by --go_out, but where exactly depends on the option go_package setting embedded within the file or on the --go_opt compiler flags, as shown below.

  protoc \
    --proto_path=<protocol-buffer-root-source-directory> \
    --go_out=<protocol-buffer-root-generated-code-directory> \
    [--go_opt=...] \
    .../<protocol-buffer-file-1>.proto .../<protocol-buffer-file-1>.proto ...

For more details on a typical Go project layout, see:

Go Project Layout

Declare Package Input Path inside the .proto Files

The Protocol Buffer compiler must know the path of the directory where to place the generated .pb.go file, relative to the --go_out directory. The recommended way to specify this path is to use the go_package option inside the .proto file. The value of the go_package option is the path of the directory, relative to the --go_out directory, where the generated file will be placed.

syntax = "proto3";

option go_package = "./somepkgpb"; // the compiler will create "./somepkgpb" in the directory indicated by --go_out

message Something { 
  ...
}

compiled with:

protoc ... --go_out=./pkg somefile.proto

will make the compiler to place the generated somefile.pb.go in ./pkg/somepkgpb. Note that the compiler expects the go_package option value to be a filesystem path, so it expects to see at least a dot (".") or a slash ("/").

The package name for the generated code file will be by default automatically inferred as the the last path fragment of the path specified with go_package option. In the example above, the package name will be somepkgpb. An explicit package name may be specified, by separating the name from the import path by a semicolon ("./somepkgpb;someothername"). However, this usage is discouraged since the package name will be derived by default from the import path in a reasonable manner.

Package names should follow Protocol Buffer package name convention.

Package Naming Convention

The package name for the Go source code files generated from .proto files are conventionally suffixed with "pb": somepkgpb. Even if it not required to use the same name for the enclosing directory, it is good practice if that happens.

Example

A typical layout that includes Protocol Buffer files that will generate code in different packages:

.
├── go.mod
├── pkg
│   ├── main
│   │    └── main.go
│   ├── somepkgpb          ┌─────────────────────────────────────────┐ 
│   │    ├── file1.pb.go  ─┤ package somepkgpb                       │ 
│   │    └── file2.pb.go   └─────────────────────────────────────────┘
│   └── someotherpkgpb     ┌─────────────────────────────────────────┐ 
│        ├── file3.pb.go  ─┤ package someotherpkgpb                  │ 
│        └── file4.pb.go   └─────────────────────────────────────────┘
└── protobuf
     ├── somepkgpb         ┌─────────────────────────────────────────┐ 
     │    ├── file1.proto ─┤ option go_package = "./somepkgpb";      │ 
     │    └── file2.proto  │ package somepkgpb;                      │
     │                     └─────────────────────────────────────────┘ 
     └── someotherpkgpb    ┌─────────────────────────────────────────┐ 
          ├── file3.proto ─┤ option go_package = "./someotherpkgpb"; │ 
          └── file4.proto  │ package someotherpkgpb;                 │
                           └─────────────────────────────────────────┘

The protoc command is:

protoc \
  --proto_path=./protobuf --go_out=./pkg \
  somepkgpb/file1.proto somepkgpb/file2.proto someotherpkgpb/file3.proto someotherpkgpb/file4.proto

Compilation

Update the necessary dependencies to go.mod with:

go get google.golang.org/protobuf/reflect/protoreflect

--go_opt

paths=import

TODO.

paths=source_relative

TODO.

module=${PREFIX}

TODO.

M${PROTO_FILE}=${GO_IMPORT_PATH}

TODO. Use Declare Package Input Path inside the .proto Files instead.

Generated Code

Translation rules:

• The dots in the Protocol Buffer package name are translated to underscore in the Go package name.
• The generated fields use a CamelCase convention, translated from snake_case.
• "protobuf" and "json" tags are also generated in the struct.

The Protocol Buffer source message:

option go_package = "./somepkgpb";
package somepkgpb;

message SomeMessage {
  int32 id = 1;
  bool  is_valid = 2;
  string name = 3;
  repeated string sample_list = 4;
}

The Go file:

package somepkgpb

type SomeMessage struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Id         int32    `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"`
	IsValid    bool     `protobuf:"varint,2,opt,name=is_valid,json=isValid,proto3" json:"is_valid,omitempty"`
	Name       string   `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
	SampleList []string `protobuf:"bytes,4,rep,name=sample_list,json=sampleList,proto3" json:"sample_list,omitempty"`
}

func (x *SomeMessage) Reset() { ... }
func (x *SomeMessage) String() string { ... }
func (*SomeMessage) ProtoMessage() {}
func (x *SomeMessage) ProtoReflect() protoreflect.Message { ... }
func (x *SomeMessage) GetId() int32 { ... }
func (x *SomeMessage) GetIsValid() bool { ... }
func (x *SomeMessage) GetName() string { ... }
func (x *SomeMessage) GetSampleList() []string { ... }

Generated enums

syntax = "proto3";
option go_package = "./somepkgpb";
package somepkgpb;

enum SomeEnum {
  UNKNOWN = 0;
  A = 1;
  B = 2;
  C = 3;
}

generates:

package somepkgpb

type SomeEnum int32

const (
	SomeEnum_UNKNOWN SomeEnum = 0
	SomeEnum_A       SomeEnum = 1
	SomeEnum_B       SomeEnum = 2
	SomeEnum_C       SomeEnum = 3
)

// Enum value maps for SomeEnum.
var (
	SomeEnum_name = map[int32]string{
		0: "UNKNOWN",
		1: "A",
		2: "B",
		3: "C",
	}
	SomeEnum_value = map[string]int32{
		"UNKNOWN": 0,
		"A":       1,
		"B":       2,
		"C":       3,
	}
)

For more details about enums, see:

enum