Go Pipelines

Internal

• Go Channels

Overview

A pipeline is a data processing pattern, aimed at processing streams of data. A pipeline consists of a series of stages. The output of a stage is connected to the input of the subsequent stage, unless they happen to be the first and last stage. The input of the first stage is the input of the pipeline. The output of the last stage is the output of the pipeline. The pipeline pattern is powerful because it offers separation of concerns: each stage can implement a different concern. Thus, stages can be modified independently of one another, the stages can be mixed and matched, arranged in fan-out and fan-out topologies, etc.

Stage

A stage has the following two properties:

1. A stage consumes and returns the same type.
2. A stage must be reified by the language so that it may be passed around.

Reification means that the language exposes a concept to the developers so they can work with it directly. Examples: functions, classes.

Stages can be combined at a higher level without modifying the stages themselves.

Stages are capable of processing elements concurrently.

Stream

A stream consists of elements, which are processed one at a time by stages.

Go Pipeline

Go Pipeline Stage

Each stage consists of an (externally) read-only output channel and an internal thread pump that invokes an element-processing function. The stream elements to be processed by the stage are read from a read-only input channel, which usually belongs to the preceding stage in the pipeline. Each element is processed by the element-processing function and the result is written on the stage's output channel.

The Stage struct looks like this:

// Stage is a structure representing a pipeline stage. It contains its own output stream, an externally read-only
// channel that is used to send the elements processed by the stage out, and a "thread pump" that pulls elements
// from the input channel, processes them by invoking the element processor function on them, and then sends them
// on the output channel. The stage is preemptable via the "done" channel.
type Stage struct {
	name             string
	done             <-chan interface{}
	in               <-chan int // the read only input stream, it is external, and usually belongs to the preceding Stage 
	out              <-chan int // the externally read only output stream, it is created by the Stage
	elementProcessor func(e int) int
}

The Stage constructor:

func NewStage(name string, done <-chan interface{}, precedingStage *Stage, elementProcessor func(e <some_type>) <some_type>) *Stage {
	out := make(chan <some_type>)
	stage := &Stage{
		name:             name,
		done:             done,
		in:               precedingStage.out, // the preceding stage also exposes a chan <some_type>
		out:              out,
		elementProcessor: elementProcessor,
	}
	// create and start the thread pump that reads from the input stream, processes elements with
	// the element processor function and writes them on the output stream
	go func() {
		// close the stage when it's done processing all input elements
		defer close(out)
		// iterate over input elements
		for e := range stage.in {
			select {
			case <-done:
				return // the stage may be forcibly closed by writing the "done" channel
			case out <- elementProcessor(e):
			}
		}
	}()
	return stage
}

// makeStage creates a "stage" - a read-only channel with a "thread pump" that pulls elements 
// from the input stage, processes them and sends them on the channel. The stage is preemptable 
// via the "done" channel.
func makeStage(done <-chan interface{}, inputStage <-chan <stage_type>, <some_other_stage_params>) <-chan <stage_type> {
	// make the stage's channel - which *is* the stage
	stage := make(chan <stage_type>)
	// create and start the thread pump that writes elements on the channel
	go func() {
		// close the stage when it's done processing all elements
		defer close(stage)
		// iterate over the input elements
		for e := range inputStage {
			select {
			case <-done:
				return // the stage may be forcibly closed via the "done" channel
			case stage <- func(e <stage_type>) <stage_type> {
				// this is the function that processes each
				// element and produces the stage's per-element
				// result, possibly by using <some_other_stage_params>
                ...
				return result
			}(e):
			}
		}
	}()
	// return the stage
	return stage
}