Go Pipelines
Internal
Overview
A pipeline is a data processing pattern, aimed at processing streams of data. A pipeline consists of a series of stages. The output of a stage is connected to the input of the subsequent stage, unless they happen to be the first and last stage. The input of the first stage is the input of the pipeline. The output of the last stage is the output of the pipeline. The pipeline pattern is powerful because it offers separation of concerns: each stage can implement a different concern. Thus, stages can be modified independently of one another, the stages can be mixed and matched, arranged in fan-out and fan-out topologies, etc.
Stage
A stage has the following two properties:
- A stage consumes and returns the same type.
- A stage must be reified by the language so that it may be passed around.
Reification means that the language exposes a concept to the developers so they can work with it directly. Examples: functions, classes.
Stages can be combined at a higher level without modifying the stages themselves.