YAML in Go: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:
=External=
* https://betterprogramming.pub/parsing-and-creating-yaml-in-go-crash-course-2ec10b7db850
=Internal=
=Internal=
* [[Go Code Examples#Code_Examples|Go Code Examples]]
* [[Go Code Examples#Code_Examples|Go Code Examples]]
* [[Go Structs]]
=Overview=
=Overview=


Line 7: Line 12:
YAML support is available in the <code>gopkg.in/yaml.v3</code> package.
YAML support is available in the <code>gopkg.in/yaml.v3</code> package.


{{Note|If the structure fields are not capitalized, they are not visible across packages, and their content will zero-ed. The structure field will be accessible, but its content will be the corresponding zero value for the type.}}
{{Warn|It seems that if the structure fields are not capitalized, they are not unmarshalled properly, even if they are used outside of the package. It is probably because <code>yaml.NewDecoder().Decode()</code> cannot see them.}}
Struct [[Go_Structs#Tags|tags]] can be used in the YAML serialization/deserialization process.
=Example=
<syntaxhighlight lang='go'>
<syntaxhighlight lang='go'>
package main
package main


import (
import (
  "fmt"
"fmt"
  "gopkg.in/yaml.v3"
"gopkg.in/yaml.v3"
  "os"
"os"
)
)


Line 28: Line 40:
//
//


type Root struct {
// Note: if the package name is "config", name the structure differently, config.Config does not work well
  Color  string  `yaml:"color"`
type Config struct {
  Details Details `yaml:"details"`
Color  string  `yaml:"color"`
Details Details `yaml:"details"`
}
}


type Details struct {
type Details struct {
  Size    int      `yaml:"size"`
Size    int      `yaml:"size"`
  Weight  float64  `yaml:"weight"`
Weight  float64  `yaml:"weight"`
  Used    bool    `yaml:"used"`
Used    bool    `yaml:"used"`
  Options []string `yaml:"options"`
Options []string `yaml:"options"`
}
}


func main() {
func main() {
config := Config{
Color: "blue",
Details: Details{
Size:    10,
Weight:  2.2,
Used:    true,
Options: []string{"light", "medium", "heavy"},
},
}
f, err := os.Create("/Users/ovidiu/tmp/test.yaml")
if err != nil { ... }
//
// Marshal recursive memory struct into a file
//
if err := yaml.NewEncoder(f).Encode(&config); err != nil { ... }
if err = f.Close(); err != nil { ... }
fmt.Printf("yaml file written and closed\n")
f, err = os.Open("/Users/ovidiu/tmp/test.yaml")
if err != nil { ... }
defer func() {
if err = f.Close(); err != nil { ... }
}()
//
// Unmarshall the file into a different memory struct
//
config2 := Config{}
err = yaml.NewDecoder(f).Decode(&config2)
    // the decoder return io.EOF on an empty YAML file, even if the file contains
    // comments. We don't want to handle this as error, but as simply "empty file"
    if errors.Is(err, io.EOF) {
      // nothing to do, the config is empty anyway
      log.Printf("empty configuration\n")
    } else {
      // handle as error
      ...
    }
fmt.Printf("%+v\n", config2)
}
</syntaxhighlight>
=Simpler Example for Automatic Unmarshalling=
Annotating the structure fields with <code>`yaml:"someField"`</code> enables automatic unmarshalling: if the YAML field name coincides with the value passed as `yaml:`, the field will be parsed and its value assigned to the structure field.
<syntaxhighlight lang='go'>
import (
"fmt"
"strings"


  root := Root{
"gopkg.in/yaml.v3"
    Color: "blue",
)
    Details: Details{
        Size:    10,
        Weight:  2.2,
        Used:    true,
        Options: []string{"light", "medium", "heavy"},
    },
  }


  f, e := os.Create("/Users/ovidiu/tmp/test.yaml")
type SomeStruct struct {
  if e != nil { ... }
Color string `yaml:"color"`
}


  //
func main() {
  // Marshal recursive memory struct into a file
yamlStr := `
  //
color: "blue"
`
s := SomeStruct{}
err := yaml.NewDecoder(strings.NewReader(yamlStr)).Decode(&s)
if err != nil {
panic(err)
}
fmt.Printf("%s\n", s.Color)
}
</syntaxhighlight>
==Providing Non-Zero Value Defaults==
The unmarshaller will only write a struct field if the YAML document contains '''something''' for that field, otherwise it will leave the struct field alone. This is how we can provide non-zero value defaults: we initialize the structure with the defaults we want, and we pass it as such to the unmarshaller.
<syntaxhighlight lang='go'>
type SomeStruct struct {
Color string `yaml:"color"`
}


  if e := yaml.NewEncoder(f).Encode(&root); e != nil { ... }
yamlStr := "" // no "color" field, the default will kick in


  if e = f.Close(); e != nil { ... }
s := SomeStruct{
  fmt.Printf("yaml file written and closed\n")
Color: "DEFAULT RED", // this is the default value
}
err := yaml.NewDecoder(strings.NewReader(yamlStr)).Decode(&s)
    ...
    assert.Equal("DEFAULT RED", s.Color)
}
</syntaxhighlight>


  f, e = os.Open("/Users/ovidiu/tmp/test.yaml")
==Custom Unmarshalling==
  if e != nil { ... }
Some times we need to turn off automatic unmarshalling and handle unmarshalling by ourselves. This is achieved by associating a <code>UnmarshalYAML(unmarshal func(interface{}) error)</code> with the struct type we want to unmarshall into. The method will be called by the decoder and it will be passed an <code>unmarshal(interface{})</code> function as argument. Invoking the <code>unmarshal()</code> on a pointer to a <code>map[string]interface{}</code> will fill the map with the parsed elements of the YAML document. Note that when we provide our custom unmarshaler, the structure fields need '''not''' be annotated with <code>`yaml:"someField"`</code>.
  defer func() {
<syntaxhighlight lang='go'>
    if e = f.Close(); e != nil { ... }
type SomeStruct struct {
  }()
Color string `yaml:"color"`
}


  //
func (s *SomeStruct) UnmarshalYAML(unmarshal func(interface{}) error) error {
  // Unmarshall the file into a different memory struct
m := map[string]interface{}{}
  //
err := unmarshal(&m)
  config2 := Root{}
if err != nil {
  if e = yaml.NewDecoder(f).Decode(&config2); e != nil { ... }
return err
 
}
  fmt.Printf("%+v\n", config2)
color, exists := m["color"]
if !exists {
s.Color = "RED" // this is how we assign a default value
} else {
s.Color = color.(string)
}
return nil
}
}
</syntaxhighlight>
</syntaxhighlight>
=Behavior when an Entire Subtree Is Missing=
If a YAML node (subtree) is missing, the corresponding <code>struct</code> field contains the [[Go_Structs#struct_Zero_Value|<tt>struct</tt> zero value]] for the corresponding <tt>struct</tt> type, so it is safe to access it. Just be mindful you will get the zero values, recursively.

Latest revision as of 16:58, 20 March 2024

External

Internal

Overview

Declare a recursive structure that matches the structure of the YAML file, and then use a YAML encoder/decoder to marshall/unmarshall data in and out.

YAML support is available in the gopkg.in/yaml.v3 package.


If the structure fields are not capitalized, they are not visible across packages, and their content will zero-ed. The structure field will be accessible, but its content will be the corresponding zero value for the type.


It seems that if the structure fields are not capitalized, they are not unmarshalled properly, even if they are used outside of the package. It is probably because yaml.NewDecoder().Decode() cannot see them.

Struct tags can be used in the YAML serialization/deserialization process.

Example

package main

import (
	"fmt"
	"gopkg.in/yaml.v3"
	"os"
)

//
// color: "blue:
// details:
//   size: 10
//   weight 2.2
//   used: true
//   options:
//     - light
//     - medium
//     - heavy
//

// Note: if the package name is "config", name the structure differently, config.Config does not work well
type Config struct {
	Color   string  `yaml:"color"`
	Details Details `yaml:"details"`
}

type Details struct {
	Size    int      `yaml:"size"`
	Weight  float64  `yaml:"weight"`
	Used    bool     `yaml:"used"`
	Options []string `yaml:"options"`
}

func main() {
	config := Config{
		Color: "blue",
		Details: Details{
			Size:    10,
			Weight:  2.2,
			Used:    true,
			Options: []string{"light", "medium", "heavy"},
		},
	}
	f, err := os.Create("/Users/ovidiu/tmp/test.yaml")
	if err != nil { ... }
	//
	// Marshal recursive memory struct into a file
	//
	if err := yaml.NewEncoder(f).Encode(&config); err != nil { ... }
	if err = f.Close(); err != nil { ... }
	fmt.Printf("yaml file written and closed\n")
	f, err = os.Open("/Users/ovidiu/tmp/test.yaml")
	if err != nil { ... }
	defer func() {
		if err = f.Close(); err != nil { ... }
	}()
	//
	// Unmarshall the file into a different memory struct
	//
	config2 := Config{}
	err = yaml.NewDecoder(f).Decode(&config2)
    // the decoder return io.EOF on an empty YAML file, even if the file contains
    // comments. We don't want to handle this as error, but as simply "empty file"
    if errors.Is(err, io.EOF) {
      // nothing to do, the config is empty anyway
      log.Printf("empty configuration\n")
    } else {
       // handle as error
       ...
    }
	fmt.Printf("%+v\n", config2)
}

Simpler Example for Automatic Unmarshalling

Annotating the structure fields with `yaml:"someField"` enables automatic unmarshalling: if the YAML field name coincides with the value passed as `yaml:`, the field will be parsed and its value assigned to the structure field.

import (
	"fmt"
	"strings"

	"gopkg.in/yaml.v3"
)

type SomeStruct struct {
	Color string `yaml:"color"`
}

func main() {
	yamlStr := `
color: "blue"
`
	s := SomeStruct{}
	err := yaml.NewDecoder(strings.NewReader(yamlStr)).Decode(&s)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s\n", s.Color)
}

Providing Non-Zero Value Defaults

The unmarshaller will only write a struct field if the YAML document contains something for that field, otherwise it will leave the struct field alone. This is how we can provide non-zero value defaults: we initialize the structure with the defaults we want, and we pass it as such to the unmarshaller.

type SomeStruct struct {
	Color string `yaml:"color"`
}

	yamlStr := "" // no "color" field, the default will kick in

	s := SomeStruct{
		Color: "DEFAULT RED", // this is the default value
	}
	err := yaml.NewDecoder(strings.NewReader(yamlStr)).Decode(&s)
    ...
    assert.Equal("DEFAULT RED", s.Color)
}

Custom Unmarshalling

Some times we need to turn off automatic unmarshalling and handle unmarshalling by ourselves. This is achieved by associating a UnmarshalYAML(unmarshal func(interface{}) error) with the struct type we want to unmarshall into. The method will be called by the decoder and it will be passed an unmarshal(interface{}) function as argument. Invoking the unmarshal() on a pointer to a map[string]interface{} will fill the map with the parsed elements of the YAML document. Note that when we provide our custom unmarshaler, the structure fields need not be annotated with `yaml:"someField"`.

type SomeStruct struct {
	Color string `yaml:"color"`
}

func (s *SomeStruct) UnmarshalYAML(unmarshal func(interface{}) error) error {
	m := map[string]interface{}{}
	err := unmarshal(&m)
	if err != nil {
		return err
	}
	color, exists := m["color"]
	if !exists {
		s.Color = "RED" // this is how we assign a default value
	} else {
		s.Color = color.(string)
	}
	return nil
}

Behavior when an Entire Subtree Is Missing

If a YAML node (subtree) is missing, the corresponding struct field contains the struct zero value for the corresponding struct type, so it is safe to access it. Just be mindful you will get the zero values, recursively.