Blue-Green Deployments with Spinnaker
External
- https://blog.spinnaker.io/introducing-rollout-strategies-in-the-kubernetes-v2-provider-8bbffea109a
- https://spinnaker.io/docs/guides/user/kubernetes-v2/rollout-strategies/#redblack-rollouts
- https://spinnaker.io/docs/guides/user/kubernetes-v2/traffic-management/
- https://www.opsmx.com/blog/spinnaker-pipeline-blue-green-strategy-with-external-versioning-and-kubernetes-deployment-object/
- https://www.opsmx.com/blog/spinnaker-integration-with-istio/
Internal
Overview
This article documents a pipeline that deploys a release in a Stage environment, waits for testing, removes the release from Stage, and deploys the same release in Prod, while preserving the previous release in Prod. The Stage and Prod are serviced by two different namespaces ('of-stage' and 'of-prod'). The application is a Helm-packaged Kubernetes application, but some of the Kubernetes resources, such as the Ingress and the Service, are created manually directly in Spinnaker, one for each namespace. Also, ReplicaSets are used instead of Deployments, because Spinnaker does not handle Deployments well. It is fine if ReplicaSet is deployed as part of the Helm chart, instead of being created manually within the Spinnaker application.
Prerequisites
Create a Spinnaker Application as described here:
Create the "of-stage" and "of-prod" namespaces.
Create the Ingresses and Services
For each namespace ('of-stage' and 'of-prod'), create an Ingress and a Service each:
of-stage
Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)
Use the following manifest (do not forget to adjust the namespace).
of-stage Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: stage
namespace: of-stage
annotations:
kubernetes.io/ingress.class: nginx
ingress.beta.kubernetes.io/sni: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 800m
nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: my-ingress.example.com
http:
paths:
- path: /stage(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: stage
port:
number: 8080
tls:
- hosts:
- my-ingress.example.com
secretName: my-secret
of-stage Service
apiVersion: v1
kind: Service
metadata:
name: stage
namespace: of-stage
spec:
type: ClusterIP
selector:
stage: 'true' # this label will be dynamically applied to the workload pods during deployment by Spinnaker
ports:
- port: 8080
name: http
targetPort: 8080
of-prod
Load Balancers → Create Load Balancer Select the appropriate "account" (Kubernetes cluster)
Use the following manifest (do not forget to adjust the namespace).
of-prod Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod
namespace: of-prod
annotations:
kubernetes.io/ingress.class: nginx
ingress.beta.kubernetes.io/sni: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 800m
nginx.ingress.kubernetes.io/proxy-read-timeout: "200"
nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addr
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: my-ingress.example.com
http:
paths:
- path: /prod(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: prod
port:
number: 8080
tls:
- hosts:
- my-ingress.example.com
secretName: my-secret
of-prod Service
apiVersion: v1
kind: Service
metadata:
name: prod
namespace: of-prod
spec:
type: ClusterIP
selector:
prod: 'true' # this label will be dynamically applied to the workload pods during deployment by Spinnaker
ports:
- port: 8080
name: http
targetPort: 8080
Create or Import Required TLS Secrets
Create the Pipeline
It will be a "provide image → deploy to stage → manual testing → manual judgement → deploy to prod" pipeline.
Name: "Stage - Manual Testing - Prod" (no "→" allowed in name)
Provide Image Tag Stage
Add stage → Evaluate Variables. Stage Name: "Provide Image Tag".
Variable name: "myapp_image_tag". Variable value: ${ parameters.myapp_image_tag }
Save Changes.
If the image tag is provided at execution as a variable, the pipeline must also be parameterized: Pipeline → Configure → Configuration → Parameters → Add parameter →
Name: myapp_image_tag
Label: no label
Required: check
Pin parameter: check
Description: "something"
Helm Chart Rendering Stage
Add stage → Bake (Manifest). Stage name: "Render Helm Chart". This stage will render the helm chart, apply the configuration overlay and overwrite the image tag. For more details, see:
It's a good idea to test the pipeline execution immediately after this stage.
Stage Deployment Stage
Add stage → Deploy (Manifest). Stage name: "Deploy in Stage". This stage will deploy the Helm chart and associate the workload with the "stage" service. For more details, see:
At this stage, we do enable Rollout Strategy Options, so we can associate the workload with the "stage" service.
⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.
Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "stage" → Traffic: Send client requests to new pods → Strategy: "Highlander". ⚠️ "Highlander" rollout strategy will take care of removing previous stage releases.
- Link to detailed explanations of what happens for each of the rollout strategies from the point of view of 1) services 2) replicasets 3) pods.
- Understand the relationship between the replicasets and the Spinnaker cluster
More details:
Manual Decision Stage
Add stage → Manual Judgement. Stage name: "Wait on Stage Testing"
Production Deployment Stage
Add stage → Deploy (Manifest). Stage name: "Deploy in Prod". This stage will deploy the same Helm chart that was tested in Stage and associate the workload with the "prod" service. For more details, see:
⚠️ If more than one service is used, ensure that the services do not share a common metadata.labels label key. If they do, there will be a "Service selector must have no label keys in common with target workload" error message when executing the stage.
Enable: "Spinnaker manages traffic based on your selected strategy" → Service(s) Namespace → Service(s): "prod" → Traffic: Send client requests to new pods → Strategy: "Red/Black". ⚠️ "Red/Black" rollout strategy will preserve previous production release.
A certain number of ReplicaSets stay around, including the pods. How to manage them? They can be manually deleted from CLUSTERS → Select ReplicaSet to be deleted → ReplicaSet Actions → Delete.
More details:
Rollback
The rollback can be implemented either by creating a parallel pipeline based on Undo Rollout (Manifest), or via a two-step UI procedure where the current "version" is disabled and the older "version" is enabled via UI.