Go Pipelines

Internal

• Go Channels

Overview

A pipeline is a data processing pattern, aimed at processing streams of data. A pipeline consists of a series of stages. The output of a stage is connected to the input of the subsequent stage, unless they happen to be the first and last stage. The input of the first stage is the input of the pipeline. The output of the last stage is the output of the pipeline. The pipeline pattern is powerful because it offers separation of concerns: each stage can implement a different concern. Thus, stages can be modified independently of one another, the stages can be mixed and matched, arranged in fan-out and fan-out topologies, etc.

Stage

A stage has the following two properties:

1. A stage consumes and returns the same type.
2. A stage must be reified by the language so that it may be passed around.

Reification means that the language exposes a concept to the developers so they can work with it directly. Examples: functions, classes.

Stages can be combined at a higher level without modifying the stages themselves.

Stages are capable of processing elements concurrently.

Stream

A stream consists of elements, which are processed one at a time by stages.

Go Pipeline

Go Pipeline Stage

Each stage consists of an (externally) read-only channel with an internal threads that writes elements on the channel. The stage (the channel) is created by a function that looks like this one:

// makeStage creates a "stage" - a read-only channel with a "thread pump" that pulls elements from the input stage, processes
// them and sends them on the channel.
func makeStage(done <-chan interface{}, inputStage <-chan <some_type>, <other_stage_specific_parameters>) <-chan <some_type> {
  // make the stage's channel - which *is* the stage
  stage := make(chan <some_type>)
  // create and start the thread that writes elements on the channel
  go func() {
    // close the stage when it's done processing all elements
    defer close(stage)
    // iterate over the input elements
    for e := range inputStage {
      select {
        case <- done:
          return // the stage may be forcibly closed via the "done" channel
        case stage <- func(e some_type) some_type {
          // this is the function that processes each
          // element and produces per-element result
          ...
          return e2
        }(e)
      }
    }
  }()
  // return the stage
  return stage
}